Error on starting a linkerd with a configuration from an standard input stream

[LukaszMarchewka]

Issue Type:

• Bug report
• Feature request

What happened:
According the documentation we should be able to launch a linkerd with a configuration from the standard input stream (https://linkerd.io/config/1.3.6/linkerd/index.html).
During starting a linkerd like here:

cat linkerd/examples/http.yaml | ./linkerd-1.3.5-32b-exec -

Output:

-XX:+AggressiveOpts -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:InitialHeapSize=33554432 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=357916672 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:+PrintCommandLineFlags -XX:+ScavengeBeforeFullGC -XX:-TieredCompilation -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseStringDeduplication 
Mar 06, 2018 10:59:43 AM com.twitter.finagle.http.HttpMuxer$ $anonfun$new$1
INFO: HttpMuxer[/admin/metrics.json] = com.twitter.finagle.stats.MetricsExporter(<function1>)
Mar 06, 2018 10:59:43 AM com.twitter.finagle.http.HttpMuxer$ $anonfun$new$1
INFO: HttpMuxer[/admin/per_host_metrics.json] = com.twitter.finagle.stats.HostMetricsExporter(<function1>)
failed for com.twitter.finagle.http.package$serverErrorsAsFailures$
Error parsing flag "": flag undefined
usage: io.buoyant.linkerd.Main [<flag>...]
flags:
  -help='false': Show this help
  -log.append='true': If true, appends to existing logfile. Otherwise, file is truncated.
  -log.async='true': Log asynchronously
  -log.async.inferClassNames='false': Infer class and method names synchronously. See com.twitter.logging.QueueingHandler
  -log.async.maxsize='4096': Max queue size for async logging
  -log.level='INFO': Log level
  -log.output='/dev/stderr': Output file
  -log.rollPolicy='Never': When or how frequently to roll the logfile. See com.twitter.logging.Policy#parse documentation for DSL details.
  -log.rotateCount='-1': How many rotated logfiles to keep around
global flags:
  -com.twitter.finagle.exp.scheduler='local': Which scheduler to use for futures <local> | <lifo> | <bridged>[:<num workers>] | <forkjoin>[:<num workers>]
  -com.twitter.finagle.liveness.sessionFailureDetector='threshold:5.seconds:2:100:4.seconds': The failure detector used to determine session liveness [none|threshold:minPeriod:threshold:win:closeTimeout]
  -com.twitter.finagle.loadbalancer.defaultBalancer='choice': Default load balancer
  -com.twitter.finagle.loadbalancer.exp.loadMetric='leastReq': Metric used to measure load across endpoints (leastReq | ewma)
  -com.twitter.finagle.loadbalancer.perHostStats='false': enable/default per-host stats.
	When enabled,the configured stats receiver will be used,
	or the loaded stats receiver if none given.
	When disabled, the configured stats receiver will be used,
	or the NullStatsReceiver if none given.
  -com.twitter.finagle.mux.gracefulShutdownEnabled='true': Graceful shutdown enabled. Temporary measure to allow servers to deploy without hurting clients.
  -com.twitter.finagle.mux.lease.exp.drainerDebug='false': GC drainer debug log (verbose)
  -com.twitter.finagle.mux.lease.exp.drainerDiscountRange='52428800.bytes,629145600.bytes': Range of discount
  -com.twitter.finagle.mux.lease.exp.drainerEnabled='false': GC drainer enabled
  -com.twitter.finagle.mux.lease.exp.drainerPercentile='95': GC drainer cutoff percentile
  -com.twitter.finagle.mux.lease.exp.nackOnExpiredLease='false': nack when the lease has expired
  -com.twitter.finagle.netty3.numWorkers='16': Size of netty3 worker pool
  -com.twitter.finagle.netty4.numWorkers='16': number of netty4 worker threads
  -com.twitter.finagle.netty4.timerTickDuration='10.milliseconds': Netty 4 timer tick duration
  -com.twitter.finagle.netty4.timerTicksPerWheel='512': Netty 4 timer ticks per wheel
  -com.twitter.finagle.netty4.trackReferenceLeaks='false': Enable reference leak tracking in Netty and export a counter at finagle/netty4/reference_leaks
  -com.twitter.finagle.serverset2.chatty='false': Log resolved ServerSet2 addresses
  -com.twitter.finagle.serverset2.client.chatty='false': Track ZooKeeper calls and responses
  -com.twitter.finagle.serverset2.dnsCacheSize='16000': Maximum size of DNS resolution cache
  -com.twitter.finagle.stats.debugLoggedStatNames='': Comma separated stat names for logging observed values (set via a -D system property to avoid load ordering issues)
  -com.twitter.finagle.stats.format='commonsmetrics': Format style for metric names (ostrich|commonsmetrics|commonsstats)
  -com.twitter.finagle.stats.includeEmptyHistograms='false': Include full histogram details when there are no data points
  -com.twitter.finagle.stats.scopeSeparator='/': Override the scope separator.
  -com.twitter.finagle.stats.statsFilter='': Comma-separated regexes that indicate which metrics to filter out
  -com.twitter.finagle.stats.statsFilterFile='': Comma separated files of newline separated regexes that indicate which metrics to filter out
  -com.twitter.finagle.stats.useCounterDeltas='false': Return deltas for counters instead of absolute values. Provides compatibility with the behavior from 'Ostrich'
  -com.twitter.finagle.thrift.maxReusableBufferSize='16384.bytes': Max size (bytes) for ThriftServiceIface reusable transport buffer
  -com.twitter.finagle.toggle.flag.overrides='': Source for the Flag-based ToggleMap.
Format is `com.yourpackage.id1=fraction1,com.yourpackage.id2=fraction2,...`
where fractions must be [0.0-1.0]
  -com.twitter.finagle.util.defaultTimerProbeSlowTasks='false': Enable reporting of slow timer tasks executing in the default timer
  -com.twitter.finagle.util.defaultTimerSlowTaskLogMinInterval='20.seconds': Minimum interval between recording stack traces for slow tasks
  -com.twitter.finagle.util.defaultTimerSlowTaskMaxRuntime='2.seconds': Maximum runtime allowed for tasks before they are reported
  -com.twitter.finagle.util.loadServiceDenied='': A deny list of implementations to ignore. Keys are the fully qualified class names. Any other implementations that are found via `LoadService.apply` are eligible to be used.
  -com.twitter.finagle.util.loadServiceIgnoredPaths='': Additional packages to be excluded from recursive directory scan
  -com.twitter.finagle.zipkin.host='localhost:1463': Host to scribe traces to
  -com.twitter.finagle.zipkin.initialSampleRate='0.001': Initial sample rate
  -com.twitter.jvm.numProcs='8.0': number of logical cores
  -com.twitter.server.announcerMap='': A list mapping service names to announcers (gizmoduck=zk!/gizmoduck)
  -com.twitter.server.promoteBeforeServing='true': Promote objects in young generation to old generation before serving requests. May shorten the following gc pauses by avoiding the copying back and forth between survivor spaces of a service's long lived objects.
  -com.twitter.server.resolverMap='': A list mapping service names to resolvers (gizmoduck=zk!/gizmoduck)
  -io.buoyant.telemetry.admin.histogramSnapshotInterval='1.minutes': Interval to snapshot histrograms
  -socksPassword='': SOCKS password
  -socksProxyHost='': SOCKS proxy host
  -socksProxyPort='0': SOCKS proxy port
  -socksUsername='': SOCKS username

What you expected to happen:
Update the doc or fix launching a linkerd
How to reproduce it (as minimally and precisely as possible):
Please, download a linkerd and launch with '-' parameter:

./linkerd -

Environment:

• linkerd/namerd version, config files: newest
• Platform, version, and config files (Kubernetes, DC/OS, etc): locally
• Cloud provider or hardware configuration: none

[rmars]

Thanks for filing this, @LukaszMarchewka! We'll take a look.

[mmrozek]

The problem is caused by fact that - is treated as a flag.
In my opinion mixing flags with the parameters without context is not the best idea (at the moment is possible to run sth like: ./linkerd-1.3.5-32b-exec -log.append=true linkerd/examples/http.yaml -log.level=INFO - and it is not clear what linkerd/examples/http.yaml means).
Maybe better solution will be to treat config path as a flag also? E.g. ./linkerd-1.3.5-32b-exec -config=linkerd/examples/http.yaml or ./linkerd-1.3.5-32b-exec -config=stdin. @rmars, what do you think about that?

[mmrozek]

@rmars, @adleong, I could take care of this issue but I need some guidance. How should it look like?

[adleong]

I think the ideal fix here would be a change to twitter/util's flag parsing so that - is not treated as a flag. I believe the offending code is here: https://github.com/BuoyantIO/twitter-util/blob/develop/util-app/src/main/scala/com/twitter/app/Flags.scala#L138

If possible, I'd like to avoid breaking backwards compatibility or introducing a second way to specify a Linkerd config.

As a workaround, ./linkerd-x.y.z -- - might work to read the config from stdin?

[mmrozek]

I could create a PR against twitter-util repo but I am still not sure if it is the best idea. In that case, we will have a - between other flags and it could be unreadable. E.g. ./linkerd-1.3.5-32b-exec -log.append=true - -log.level=INFO. Maybe the better solution is to use an alphanumeric alias? If we use stdin instead of - it will be possible to reasoning about this argument and we will preserve backwards compatibility.
Btw, your workaround is working.