tag:blogger.com,1999:blog-35226889124857082832024-03-28T00:28:25.032-07:00Wandering Quandarieshihihttp://www.blogger.com/profile/14260514478423290823noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-3522688912485708283.post-82991440711894618002015-11-19T09:59:00.000-08:002015-11-19T10:01:21.710-08:00Tar b64 Copy PastaLets say you might want to copy a few different files(maybe with non-matching names in a directory) from one host to another. scp might take a few commands, but if you can squeeze them all into a small tarball this trick I learned from an old mentor is pretty handy.<br />
<br />
Compress and tarball files, and pipe the output into base64 which will get dumped to your terminal.<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">tar -czf - ./filesyouwant* | base64</span></blockquote>
<br />
Copy only the base64 output characters into your host's clipboard.<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">cat | base64 -d | tar -xzf -</span></blockquote>
Execute this command to start cat waiting for the character input. Then paste the base64 previously copied.<br />
Once all the base64 has finished pasting, hit Enter, then Ctrl^d to release cat. The pasted base64 will get decoded and tar will decompress the binary data into files of your current working directory!<br />
<br />
<br />
<br />
<br />hihihttp://www.blogger.com/profile/14260514478423290823noreply@blogger.com0tag:blogger.com,1999:blog-3522688912485708283.post-4608788870470087992014-11-06T21:41:00.005-08:002014-11-06T21:50:42.470-08:00Setting up an Insecure Docker Registry<span style="font-family: 'Trebuchet MS', sans-serif;">Running anything in an insecure mode is always dangerous. However, if the goal is to simply test something out or run in a secured environment, it can be useful. Thus was my use case to learn about using the </span><a href="https://github.com/docker/docker-registry" style="font-family: 'Trebuchet MS', sans-serif;" target="_blank">Docker Registry</a><span style="font-family: 'Trebuchet MS', sans-serif;"> and for speed of not bothering with SSL Certificates, run it in an insecure mode. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">I'm not going to cover working with <a href="https://www.docker.com/" target="_blank">Docker</a> in general, just setting up an insecure registry, head over to the <a href="https://docs.docker.com/" target="_blank">documentation</a> first and to learn more. I'm just going to journal this problem so hopefully no one has to waste time figuring it out.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">There are <a href="http://www.activestate.com/blog/2014/01/deploying-your-own-private-docker-registry" rel="nofollow" target="_blank">many</a>, <a href="http://blog.octo.com/en/docker-registry-first-steps/" target="_blank">numerous</a>, <a href="https://blog.codecentric.de/en/2014/02/docker-registry-run-private-docker-image-repository/" rel="nofollow" target="_blank">blog posts</a> about setting up Docker's Registry(properly), and most go over setting up some sort of authentication(recommend). However if the risky insecure route is fine, there's a slight hicup I found which was rather opaque to solve.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">So let's follow the basics of getting a localized Docker Registry running.</span><br />
<br />
<ol>
<li><span style="font-family: Trebuchet MS, sans-serif;">Pull the <a href="https://registry.hub.docker.com/_/registry/" target="_blank">registry image</a> from docker hub</span></li>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">docker pull registry</span></li>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Run the container with local environment(quieter output than default dev)</span></li>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">docker run -d -p 5000:5000 --name registry registry </span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">A </span><span style="font-family: Courier New, Courier, monospace;">docker ps</span><span style="font-family: Trebuchet MS, sans-serif;"> should now show it running</span></li>
<li><span style="font-family: Trebuchet MS, sans-serif;">You can check if it's running but hitting localhost:5000/ on your web browser, which should return:</span></li>
<ul>
<li><pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: Trebuchet MS, sans-serif;">"docker-registry server (dev) (v0.8.1)"</span></pre>
</li>
</ul>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Push an image to the repo prefixed with the registries address. Try a basic `ubuntu` image to the registry on your local machine: localhost:5000</span></li>
<ol>
<li><span style="font-family: Trebuchet MS, sans-serif;">One quick aside, the way Docker will switch from using the default DockerHub api is to prefix the image name with the new registry address. So tag the basic 'ubuntu' image with 'localhost:5000/ubuntu'.</span></li>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">docker tag ubuntu localhost:5000/ubuntu </span><span style="font-family: Trebuchet MS, sans-serif;"> </span></li>
<li><span style="font-family: Courier New, Courier, monospace;">docker images</span><span style="font-family: Trebuchet MS, sans-serif;"> to verify it worked</span></li>
</ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Then push the tagged image</span></li>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">docker push localhost:5000/ubuntu</span></li>
</ul>
</ol>
</ol>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">And here comes the error...</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<br />
<span style="font-family: Courier New, Courier, monospace;">Error response from daemon: Invalid registry endpoint https://localhost:5000/v1/: Get https://localhost:5000/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry localhost:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/localhost:5000/ca.crt</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://i1207.photobucket.com/albums/bb476/ecklehmann/wat.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://i1207.photobucket.com/albums/bb476/ecklehmann/wat.gif" height="192" width="320" /></a></div>
<span style="font-family: Trebuchet MS, sans-serif;"> </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">I though this was running in an open dev mode?</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">There are multiple environments which the registry can run under with varying settings, they can be seen in the <a href="https://github.com/docker/docker-registry/blob/master/config/config_sample.yml" target="_blank">provided config file</a>. However after digging through blogs, I eventually stumbled across this<a href="https://github.com/docker/docker/issues/8887" target="_blank"> Github issue</a> which flipped the light switch.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">The error comes from the Docker daemon running in the normal mode which by default requires HTTPS for communicating with Registry APIs. The 'daemon' is the the local machine's and not the registry's server (should have been drinking more coffee). </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h2>
<span style="font-family: Trebuchet MS, sans-serif;">Solution</span></h2>
<span style="font-family: Trebuchet MS, sans-serif;">Shut down your docker daemon. On debian based distros:</span><span style="font-family: Courier New, Courier, monospace;"> </span><br />
<br />
<ul>
<li><span style="font-family: 'Courier New', Courier, monospace;">sudo service docker stop</span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">Run the docker daemon with the insecure flags: </span><br />
<br />
<ul>
<li><span style="font-family: 'Courier New', Courier, monospace;">sudo docker -d --insecure-registry localhost:5000</span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">Start the `registry` container since it was stopped when the old daemon shutdown. </span><br />
<br />
<ul>
<li><span style="font-family: 'Courier New', Courier, monospace;">docker start registry</span></li>
</ul>
<br />
<span style="font-family: Trebuchet MS, sans-serif;">and..</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">>>>>>>>>>>>>sudo docker push localhost:5000/ubuntu<br />The push refers to a repository [localhost:5000/ubuntu] (len: 1)<br />Sending image list<br />Pushing repository localhost:5000/ubuntu (1 tags)<br />511136ea3c5a: Image successfully pushed<br />d497ad3926c8: Image successfully pushed<br />ccb62158e970: Image successfully pushed<br />e791be0477f2: Image successfully pushed<br />3680052c0f5c: Image successfully pushed<br />22093c35d77b: Image successfully pushed<br />5506de2b643b: Image successfully pushed<br />Pushing tag for rev [5506de2b643b] on {http://localhost:5000/v1/repositories/ubuntu/tags/latest}</span></blockquote>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;">Now you can mess with an unsecured registry!</span></div>
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<span style="font-family: Trebuchet MS, sans-serif;">Hopefully this post might save people some time figuring out what is wrong. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Cheers!</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><a href="https://twitter.com/joshroppo" target="_blank">@joshroppo</a></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<div>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span></div>
<br />hihihttp://www.blogger.com/profile/14260514478423290823noreply@blogger.com0tag:blogger.com,1999:blog-3522688912485708283.post-59782390535508365222014-05-06T23:41:00.000-07:002014-05-09T10:56:19.599-07:00Write the Docs 2014 NA: Volunteering Tales!<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisAPkW6c-gQBBk8P7tvkNy64hQejKkSACqMI3tPmSLlPdRfEtV8aRCudIu8jQlCqJKjZax213HKV0jG7U2d82i1-2jEqSKmQM8LyM1M3Fs1YrF2MRBrXABMaioHDyvjmufJ7vqgt0eXQY/s1600/PANO_20140505_085949.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisAPkW6c-gQBBk8P7tvkNy64hQejKkSACqMI3tPmSLlPdRfEtV8aRCudIu8jQlCqJKjZax213HKV0jG7U2d82i1-2jEqSKmQM8LyM1M3Fs1YrF2MRBrXABMaioHDyvjmufJ7vqgt0eXQY/s1600/PANO_20140505_085949.jpg" height="152" width="640" /></a></div>
*ROUGH DRAFT* (Taking hints from all the great speakers and just getting something written!)<br />
Getting to be part of <a href="http://conf.writethedocs.org/na/2014/index.html" target="_blank">NA Write the Docs</a> conference was a really fun experience. It was exhausting and a little stressful from time to time but things rolled smoothly and overall seemed to be a great success!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRtLoy22wjAcDjPWk75ZXnZ5dPQ6csOxuROxtk_42LvTpvP8_CjJXlI-qcoEuiLu7UCFBtieEPXecvg7ZO61pDK6lKtNafV4nuEzuTNZZBzTTG9vpfoKsq68IO0RK47fx0e-Ri5e8pYwg/s1600/PANO_20140505_152629.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRtLoy22wjAcDjPWk75ZXnZ5dPQ6csOxuROxtk_42LvTpvP8_CjJXlI-qcoEuiLu7UCFBtieEPXecvg7ZO61pDK6lKtNafV4nuEzuTNZZBzTTG9vpfoKsq68IO0RK47fx0e-Ri5e8pYwg/s1600/PANO_20140505_152629.jpg" height="108" width="320" /></a></div>
<br />
<br />
Volunteering role: my main task was to wrangle speakers by making sure they're checked in, and knew where they were needed to be. The Crystal Ballroom's musically charged Green Room was the quiet prep room for speakers. An email was sent to the speakers of its availability. However, not all of them saw email/found the room, so finding and checking them in became the most difficult task. Once the speaker was found by any of the conference organizers things calmed down since we knew they were at least there. However I had some frantic searches comparing names with their fuzzy head shots to track people down. In some cases there were no head shots and none of the organizers knew what the speaker looked like(uncommon worst case).<br />
<br />
First day was the craziest as with doing anything for the first time. Often desperately walking through the crowd attempting to identify speakers.<br />
<blockquote class="tr_bq">
<i>I apologize to all the attendees who were weirded out by my awkward glances at their name badge.</i></blockquote>
<ul style="margin-left: 40px;">
<li> Side note: I wish the badge lanyards were shorter to make gawking names easier.. </li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc7t3izghyap8iX2oZNhlxEZ2TRYOL-J6q-Cgt03HZ_tIFqfOERPrS93Yyn6S0rwKrlHNIBwTlr0_we84YjVBp_IvoWfGo-kBtIWQo7hY99iaJ6HNr5dRw-5nC1Vs8NnLhmGUprQ5mQ5A/s1600/PANO_20140506_181748.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc7t3izghyap8iX2oZNhlxEZ2TRYOL-J6q-Cgt03HZ_tIFqfOERPrS93Yyn6S0rwKrlHNIBwTlr0_we84YjVBp_IvoWfGo-kBtIWQo7hY99iaJ6HNr5dRw-5nC1Vs8NnLhmGUprQ5mQ5A/s1600/PANO_20140506_181748.jpg" height="244" width="640" /></a></div>
Once a speaker was found, the general procedure was to..<br />
<ul>
<li>Make sure the speaker knew of the Green Room('s location) and its quietness </li>
<li>Give them their gifts: Coffee, Chocolate, and Hoodie! </li>
<li>If they chose to prep in the Green Room</li>
<ul>
<li> Act as their talk-time wake-up-call</li>
<li>Introduce speaker to the AV crew to ensure no surprises</li>
</ul>
<li> Generally make sure they were informed about what is going on and inform of any delays.</li>
</ul>
All of the speakers were incredibly kind and most fairly happy to find the quiet of the Green Room to collect their thoughts. It wasn't completely necessary to coral the speakers but knowing they were ready and knowing that they knew when they needed to be on stage was comforting to all the organizers. It also ensured that speaker transitions went fairly smooth.<br />
<br />
Potential improvements, biggest would be easily tracking down the speakers and getting them checked in. A few ideas <a href="https://twitter.com/ruthlovescake" target="_blank">Ruth</a> and I bounced back and fourth a little bit while the conference was winding down:<br />
<ul>
<li>Having mobile contact info for all the Speakers</li>
<li>Requiring mug shots of all speakers..</li>
<li>Different name-badges</li>
<li>At sign in keep their badges aside during registration so one of the organizers can get them informed/checked-in and get mental facial recognition established.</li>
</ul>
Write the Docs was the first formal tech conference I've been involved with and I'm glad I volunteered. I have helped out with some <a href="http://pigsquad.com/" target="_blank">PIGSquad</a> Game-Jams in the past but those were less organizationally challenging and smaller scale. <br />
<br />
One of the biggest surprises for me was how polite and generally excited all the conference attendees were! Everyone smiled while passing in the hallway and the atmosphere was exceptionally warm and inviting. At all the tech events I've been to in the past there are generally a few difficult individuals and zones of silence which develop where no one is interested in talking. Not the case at WtD! It almost seemed that if there were magical universal time allowances for everyone, the conference and discussion would have gone on for days longer!<br />
<br />
<a href="https://twitter.com/ericholscher" target="_blank">Eric</a>, <a href="https://twitter.com/thoward37" target="_blank">Troy</a>, and <a href="https://twitter.com/ruthlovescake" target="_blank">Ruth</a> definitely have found a group of people who are the unicorn merging of technical writers and developer who care about documenting the intellectual achievements of our Information Age and haven't had a place to congregate before. That might partly explain the enthusiasm of the attendees. All of the organizers deserve serious props for creating such an inviting atmosphere and great content! <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-wak0GH3i77pnYFcN5sASzneVromkKaHpuuz3z1F-YTnU8UC4rTXU7Jx3I6ibrxDb9VYXlgLTA_MSsVPJtvc52BM_F2RT0O7nRLDPoLLgfRjsMT4Q3MvRYN5m9bH3yy27fNh7sZxEpIg/s1600/IMG_20140506_170549.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-wak0GH3i77pnYFcN5sASzneVromkKaHpuuz3z1F-YTnU8UC4rTXU7Jx3I6ibrxDb9VYXlgLTA_MSsVPJtvc52BM_F2RT0O7nRLDPoLLgfRjsMT4Q3MvRYN5m9bH3yy27fNh7sZxEpIg/s1600/IMG_20140506_170549.jpg" height="300" width="400" /></a></div>
<br />
<br />
To be quite honest I wasn't insanely passionate about documentation before the conference. My main opinion previously was from my former job which was: it really sucks when there's no good docs on systems. On my last day I stayed up till 4AM to document the projects I was leaving behind, I didn't want to leave a vacuum of knowledge like I had already experienced. Post-conf, seeing all of the great talks definitely has strengthened my appreciation for documentation. Wherever I end up working next, I hope we write the docs in an efficient and appreciative fashion!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfONP_SRMHh4PkwwIwJ9z3WaiANMNmPFHkaAjFh5WcuEpl6fae2-5GwU8ZrOv3FBAjIJO5_RTjEfto3kHbDrFHT82HWTdCeKY9Kz3WfiYuIwziORrTU8kR0g-53WhtJYPKT8ZeHATp7cw/s1600/IMG_20140506_181431.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfONP_SRMHh4PkwwIwJ9z3WaiANMNmPFHkaAjFh5WcuEpl6fae2-5GwU8ZrOv3FBAjIJO5_RTjEfto3kHbDrFHT82HWTdCeKY9Kz3WfiYuIwziORrTU8kR0g-53WhtJYPKT8ZeHATp7cw/s1600/IMG_20140506_181431.jpg" height="400" width="300" /></a></div>
<div style="text-align: center;">
<a href="https://twitter.com/nikblanchet" target="_blank">Nik</a> ensuring Write The Docs legacy remains on the Crystal legends board!</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5di1WZ5vrX6-mu9XF5X2q0_83NsAArLRxJNYSWK4j7lk7KPL9DFnU0Ew3FaYzz9XTuJvne-Z9El4nEKoprnG85lzgR24wXtu32ECAs559_vz0RoMUxJ5_vZfGKs44M2nzytKKewHMvnE/s1600/IMG_20140506_181501.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5di1WZ5vrX6-mu9XF5X2q0_83NsAArLRxJNYSWK4j7lk7KPL9DFnU0Ew3FaYzz9XTuJvne-Z9El4nEKoprnG85lzgR24wXtu32ECAs559_vz0RoMUxJ5_vZfGKs44M2nzytKKewHMvnE/s1600/IMG_20140506_181501.jpg" height="300" width="400" /></a></div>
<div style="text-align: center;">
Great times!</div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
Hopefully when my brain has recovered I'll add more to this post as great moments trickle through my mind...</div>
hihihttp://www.blogger.com/profile/14260514478423290823noreply@blogger.com0tag:blogger.com,1999:blog-3522688912485708283.post-69035109644584526012014-04-25T11:10:00.007-07:002014-04-25T11:14:28.141-07:00Python to Scala: Virtualenvs to sbt for project management<div class="tr_bq">
<span style="font-family: Verdana,sans-serif;">At <a href="http://pdxscala.org/" target="_blank">PDX Scala</a> on 2014/4/9, <a href="https://twitter.com/tlockney" target="_blank">Thomas</a> gave a great introduction to using <a href="http://www.scala-sbt.org/" target="_blank">sbt</a> for simple to complex project management. Most of my experience dealing with significant dev environments comes from the Python world using <a href="https://pypi.python.org/pypi/virtualenv" target="_blank">Virtualenv</a> and its handy <a href="https://pypi.python.org/pypi/virtualenvwrapper" target="_blank">wrapper</a>.</span></div>
<h2>
<span style="font-family: Verdana,sans-serif;">sbt big takeaways:</span></h2>
<ul>
<li><span style="font-family: Verdana,sans-serif;">Fully unified tool built in Scala for project management and development</span></li>
<li><span style="font-family: Verdana,sans-serif;">Tilda operators give scripting language flexibility to compiled Scala</span></li>
<li><span style="font-family: Verdana,sans-serif;">Very similar to Python's Virtualenv-Pip tools, but unified into a single tool</span></li>
<li><span style="font-family: Verdana,sans-serif;">(Learned the hard way) Many of the simplicities of Python/Interpreted languages don't translate to the Scala/JVM world. The sbt documentation expects a certain amount of JVM domain knowledge which I had long forgotten.</span></li>
<ul>
<li><span style="font-family: Verdana,sans-serif;">Ergo: Configuration is more difficult than Virtualenv's </span></li>
</ul>
</ul>
<span style="font-family: Verdana,sans-serif;">Going further; I hope to </span><span style="font-family: Verdana,sans-serif;"><span style="font-family: Verdana,sans-serif;">compare and contrast the two tool sets to gain a better understanding of both. All of what I say about the sbt side of things is subject to immense salt and newbie understanding of the Scala/Java world. Please respond with corrections, constructive criticism, and improvements!</span></span><br />
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: Verdana,sans-serif;">If there's one area of lacking in my understanding of the Scala world, is the legacy of Java and all the paradigms of managing the JVM. </span> </span><br />
<span style="font-family: Verdana,sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Verdana,sans-serif;">Setup Project:</span></h3>
<span style="font-family: Verdana,sans-serif;"><b>Virtualenv:</b> Use virtualenvwrapper to initialize environment, then create directories for project.</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ mkvirtualenv venv</span><br />
<span style="font-family: "Courier New",Courier,monospace;">(venv)$ cd projectdir </span><br />
<span style="font-family: "Courier New",Courier,monospace;">(venv)$ mkdir projectsrc</span><br />
<span style="font-family: "Courier New",Courier,monospace;">(venv)$ touch projectsrc/__init__.py</span><br />
<span style="font-family: "Courier New",Courier,monospace;">(venv)$ echo 'print("hihi!")' > projectsrc/hihi.py</span><br />
<span style="font-family: "Courier New",Courier,monospace;">(venv)$ python -m projectsrc.hihi </span></blockquote>
<br />
<ul>
</ul>
<span style="font-family: Verdana,sans-serif;"><b>sbt:</b> In project's directory, activate sbt tool and <a href="http://www.scala-sbt.org/release/docs/Getting-Started/Hello.html#create-a-project-directory-with-source-code" target="_blank">create directory structure to match what is expected by sbt</a>:</span><br />
<blockquote class="tr_bq">
<ul class="simple">
<li><span style="font-family: Verdana,sans-serif;">Sources in the base directory</span></li>
<li><span style="font-family: Verdana,sans-serif;">Sources in <span class="pre">src/main/scala</span> or <span class="pre">src/main/java</span></span></li>
<li><span style="font-family: Verdana,sans-serif;">Tests in <span class="pre">src/test/scala</span> or <span class="pre">src/test/java</span></span></li>
<li><span style="font-family: Verdana,sans-serif;">Data files in <span class="pre">src/main/resources</span> or <span class="pre">src/test/resources</span></span></li>
<li><span style="font-family: Verdana,sans-serif;">jars in <span class="pre">lib</span></span><span style="font-family: Verdana,sans-serif;"></span></li>
</ul>
</blockquote>
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ cd projectdir</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ touch build.sbt #sbt base config</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ mkdir -p src/main/scala</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ echo 'object Hi { def main(args: Array[String]) = println("hihi!") }' > src/main/scala/hw.scala</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ sbt</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ run</span></blockquote>
<span style="font-family: Verdana,sans-serif;">Memory management voodoo from Thomas; in the home directory create an ~/.sbtconfig file and add memory management flags for execution:</span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"</span></blockquote>
<span style="font-family: Verdana,sans-serif;">Apparently sbt can start to eat up a lot of memory if left running for a long time. Don't know details, just standing on larger shoulders here.</span><br />
<ul>
</ul>
<h3>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Project Customization</span></span></span></h3>
<h4>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Dependency Management</span></span></span></h4>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><b>Virtualenv/Python:</b>
Using virtualenv pip to install all the necessary libraries makes it
easy to export the versioned dependencies of a project:</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">(venv)$ pip freeze > projectdir/requirements.txt</span></blockquote>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">When
cloning a codebase, assuming the owner has been keeping the
requirements file up to date, a new user can use the file to mirror
install all the necessary dependencies.</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><span style="font-family: "Courier New",Courier,monospace;">(venv)$ pip install -r projectdir/requirements.txt</span> </span></span></span></blockquote>
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Furthermore properly configuring the <span style="font-family: "Courier New",Courier,monospace;">projectdir/setup.py</span> file ,which runs under <a href="https://pypi.python.org/pypi/setuptools" target="_blank">setuptools</a> to build installable artifacts for deployment, should also contain a manifest of required libraries.</span></span></span><br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><br /></span></span></span>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><b>sbt:</b>
Like seemingly most jvm systems, configuration runs dark and deep. sbt
is fairly clean but can become very powerful if the dev is
knowledgeable enough. I'm only going to cover the basic build.sbt file. Deeper documentation to use the the Build.scala files can be found <a href="http://www.scala-sbt.org/release/docs/Getting-Started/Full-Def.html" target="_blank">here</a>(maybe another blog post).</span></span></span><br />
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Simple library requirements: use <a href="http://search.maven.org/" target="_blank">Maven central repository</a> to find the library, pull up its Artifact Details page, and in Dependency Information; copy the 'Scala SBT' definition and add it to the line separated build.sbt. eg:</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">libraryDependencies += "com.typesafe.slick" % "slick_2.10" % "2.0.1"</span></blockquote>
<span style="font-family: Verdana,sans-serif;">The following $compile will resolve the dependencies</span><span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">.</span></span></span><br />
<span style="font-family: Verdana, sans-serif;">Additionally there are ways of adding dependencies via sbt's CLI; which can be found TODO:here.</span><br />
<h3>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"> </span></span></span></h3>
<h4>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Environment Variables </span></span></span></h4>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><b>Virtualenv:</b> Personal taste, using the virtualenv setup script <span style="font-family: "Courier New",Courier,monospace;">postactivate</span> for loading any environment variables. Mileage will vary and virtualenvs allow several points of entry for customization. I prefer to tie the env-vars to the virtual env so if you want to check something in the REPL there's no requirement to be in the projects directory as if using <a href="https://github.com/kennethreitz/autoenv" target="_blank">autoenv</a>(although it is a cool tool).</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">cd ~/.virtualenvs/<project_name>/bin</project_name></span> <br />
<span style="font-family: "Courier New",Courier,monospace;">vi postactivate</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Verdana,sans-serif;">Write:</span> export POSTGRESPASS="123456"</span></blockquote>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"> </span></span></span><br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><b>sbt:</b> Figuring out how to get environment variables into sbt runtime became my White Whale.. Ultimately it simply required a deeper understanding of sbt's internals and settings management. Along with realizing that the 'envVars' setting is only applied to runtimes where the compiled process is forked. </span></span></span><br />
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Ultimately while Environment Variables are often used in Python systems for defining sensitive information or development state. Conversely<b> the JVM ecosystem prefers compilation or runtime configuration (arguments/flags) instead of using system definitions like environment variables which interpreted languages tend to favor.</b> Via the freenode #scala channel; <a href="https://twitter.com/tpolecat" target="_blank">tpolecat</a> nicely confirmed that the general jvm practice is to specify vm runtime system property configuration via CLI arguments is common practice(I trust his opinion).</span></span></span><br />
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">HOWEVER, if there is still a wild need to specify environment variables for runtime, sbt recently added support for it(with exceptions). </span></span></span><br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">In the declarative build.sbt file:</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">fork := true<br /><br />envVars ++= Map{"ENVIRONMENT_DEF" -> "dev"}</span></blockquote>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Caveat/"fork := true" explanation: the "envVars" setting is only applied to VMs which have been forked from the standard sbt process. Then envVars setting is not loaded into the sbt process and therefore can't be referenced in the 'console' REPL.</span></span></span><br />
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">The previous build.sbt definition will map "dev" to "ENVIRONMENT_DEF" and can be referenced in a forked vm with:</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">System.getenv("ENVIRONMENT_DEF")</span></blockquote>
<br />
<h3>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">REPL</span></span></span></h3>
<span style="font-family: Verdana,sans-serif;"><b>Virtualenv</b> links to the project python binary which is configured to use all of the libraries which have been installed by the localized pip. This can include a nicer REPL like iPython which will be scoped to virtualenv. </span><br />
<span style="font-family: Verdana,sans-serif;"><br /></span>
<span style="font-family: Verdana,sans-serif;"><b>sbt </b>has a '<span style="font-family: "Courier New",Courier,monospace;">console</span>' command which acts like the normal '<span style="font-family: "Courier New",Courier,monospace;">scala</span>' REPL. When used the interpreter runs under the project's configuration and defined dependencies are accessible. One caveat mentioned earlier is that the console exists in the same vm process as sbt, so changes set for when forked will not populate to the REPL.</span><br />
<br />
<h3>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Packaging</span></span></span></h3>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><b>Python:</b> Once a proper <a href="http://pythonhosted.org//setuptools/setuptools.html#developer-s-guide" target="_blank">setuptools setup.py definition</a> has been created, producing the project artifact to install is simple.</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">python setup.py sdist</span></span></span></blockquote>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">Will tar up all of the specified files into a source distribution artifact which can be installed by pip remotely on the server with <a href="http://www.fabfile.org/" target="_blank">fabric</a>.</span></span></span><br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><br /></span></span></span>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">sbt:Assuming that the library dependencies are properly specified, sbt will build a jar file very simply with the '<span style="font-family: "Courier New",Courier,monospace;">package</span>' command.</span></span></span><br />
<blockquote class="tr_bq">
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;"><span style="font-family: "Courier New", Courier, monospace;">> package</span> </span></span></span></blockquote>
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">There is also tooling similar to fabric which can handle deployment under the 'publish' operation, obviously this requires more configuration.</span></span></span><br />
<br />
<br />
<span style="font-family: Verdana,sans-serif;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: Verdana, sans-serif;">This is just my simple overview of how Virtualenvs and sbt compare, there's more to cover but I think these are good basics to start with. Please comment to point out any inaccuracies or things I might have missed. </span></span></span>hihihttp://www.blogger.com/profile/14260514478423290823noreply@blogger.com0Portland, OR, USA45.5234515 -122.676207145.1675755 -123.3216541 45.8793275 -122.0307601tag:blogger.com,1999:blog-3522688912485708283.post-75085885966246913852014-02-15T04:21:00.000-08:002014-02-15T04:21:01.562-08:00Docker and X: A match made with filesystems<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://www.docker.io/">Docker</a> has been generating a lot of hype recently, and for good reason! A lightweight alternative to VMs which can be version controlled and sent strait from development to production! What's not to love?</span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">Running a web servers are fairly strait forward applications and generally well battle tested. Something that isn't stable and generally fought with quirks is the X desktop. So lets look into whether Docker can support the mess that is X.</span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">When attempting to install an X desktop in a Docker container there is an issue, Docker doesn't have perfect access to all the core system devices which X needs:</span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: #cccccc;"><span style="background-color: #666666;"><span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">Creating fuse device...<br />mknod: `fuse-': Operation not permitted<br />makedev fuse c 10 229 root root 0660: failed<br />chown: cannot access `/dev/fuse': No such file or directory<br />dpkg: error processing fuse (--configure):<br /> subprocess installed post-installation script returned error exit status 1<br />Processing triggers for libc-bin ...<br />ldconfig deferred processing now taking place<br />Errors were encountered while processing:<br /> fuse<br />E: Sub-process /usr/bin/dpkg returned an error code (1)</span></span></span></span></span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">This error is thrown whenever attempting to install xfce or lxde. Consequently I did some googling to skirt the issue of installing full X on Docker and couldn't come up with much. Using X desktops and Docker is obviously not it's main use case, so it's somewhat expected. However I decided to search for what I was trying to accomplish, running Selenium in a Docker container; this has been done, and quite nicely. </span><br />
<br />
<a href="https://index.docker.io/u/vvoyer/docker-selenium-firefox-chrome/"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">Vvoyer's Docker Selenium Container</span></a><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">The solution is to use <a href="http://en.wikipedia.org/wiki/Xvfb">Xvfb</a>, which completely bypasses the need for a full graphical stack and allows selenium to run quietly in buffers. So assuming you're confident in your selenium procedures, everything should proceed as usual.</span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://twitter.com/zeroload">Vincent Voyer</a> made a nice write up of design and usage of the container which can be found on his blog: <a href="http://www.hnwatcher.com/r/586459/Easy-selenium-chrome-Firefox-installs-with-Docker">Easy-selenium-chrome-Firefox-installs-with-Docker</a></span><br />
<br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">There are a few things I would change in regards to using Chromium instead of Chrome, but otherwise it's a solid solution and baseline for working with X in Docker containers. Hopefully I'll be able to dig deeper into the /dev/fuse error and understand what the real problem is. </span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span>hihihttp://www.blogger.com/profile/14260514478423290823noreply@blogger.com0