terriyu@/media/blog

New mobile friendly theme

2016-04-09T00:00:00-04:00

I noticed that my blog is not mobile friendly, so I changed my Pelican theme to use Bootstrap version 3, which has a responsive design. I am now using pelican-bootstrap3 with the Bootswatch United theme.

The United theme is inspired by Ubuntu and appears to use similar colors and fonts. This seemed appropriate, given my support of free open source software.

I recompiled the Bootswatch United theme to increase the base font size from 14 pt to 18 pt. Maybe some people think that's big, but I like large text! The article titles are kind of big on my phone though.

To recompile the CSS for a Bootswatch theme, follow the instructions here. The changes I made are shown in this diff:

diff --git a/united/variables.less b/united/variables.less
index 1062bbb..15a399a 100644
--- a/united/variables.less
+++ b/united/variables.less
@@ -48,12 +48,12 @@
 @font-family-monospace:   Menlo, Monaco, Consolas, "Courier New", monospace;
 @font-family-base:        @font-family-sans-serif;

-@font-size-base:          14px;
+@font-size-base:          18px;
 @font-size-large:         ceil((@font-size-base * 1.25)); // ~18px
 @font-size-small:         ceil((@font-size-base * 0.85)); // ~12px

-@font-size-h1:            floor((@font-size-base * 2.6)); // ~36px
-@font-size-h2:            floor((@font-size-base * 2.15)); // ~30px
+@font-size-h1:            floor((@font-size-base * 2.22)); // ~36px
+@font-size-h2:            floor((@font-size-base * 1.89)); // ~30px
 @font-size-h3:            ceil((@font-size-base * 1.7)); // ~24px
 @font-size-h4:            ceil((@font-size-base * 1.25)); // ~18px
 @font-size-h5:            @font-size-base;

In short, make changes to variables in variables.less and then recompile to get the new CSS files.

Mostly, I was able to plug in the Bootstrap 3 theme with minimal changes. The only thing I haven't gotten to work is the Tipue Search plugin.

Becoming a new contributor to open source through Outreach Program for Women and OpenStack

2013-07-31T00:00:00-04:00

Attracting new contributors and making projects accessible to newcomers is an important issue to many open source organizations.

I will describe my experiences becoming a new contributor to open source. I hope that this will be useful to 1) people trying to improve the open source newcomer experience and 2) people who have never worked on open source and are interested in becoming contributors.

Initial thoughts

I've always wanted to develop software, and my friends told me that participating in open source is a good way to get experience. About a year ago, I started looking into open source projects.

During this time, I thought about what kind of open source project I'd like to join. I came up with a few criteria.¹

What is the size of the project and is there a lot of activity?

If it's too small, there might not be enough opportunity for interaction. If it's too large, I might be overwhelmed.
Am I interested in the goals of the project?

I like number crunching and am interested in learning more about core infrastructure like cloud computing and networking.
Is there any formal effort to welcome newbies, like a mentoring group, newbies mailing list, etc?

I never felt like I could just jump in and immediately contribute, at least not when I was first getting started.
What programming language and tools does the project use?

I prefer to work in a language I already know and enjoy using, but I might be interested in learning a new tool if it sounds cool.

I had experience using FOSS software like Emacs, MikTeX, and Mercurial, but the projects that had the greatest appeal to me were the programming language Python and associated scientific libraries NumPy/SciPy. I'm excited about scientific computing, so I was interested in developing tools that I had previously used in scientific research.

I found out that Python has a mentoring mailing list, so I set my sights on contributing to the Python Core. But then I discovered the Outreach Program for Women and decided to apply for a free open source software (FOSS) internship there.

Applying to Outreach Program for Women

The Outreach Program for Women² is unique in that the application process itself is a simulation of the actual internship. Applicants spend their time during the application period learning the tools and culture of FOSS, so that they can make their first contributions.

Therefore, applying to the Outreach Program for Women became my first experience with FOSS as a contributor. This isn't exactly the same as just "walking" into an open source project off the internet, since a program inherently has more structure and organization. But I still think much of my experience is relevant to the "open source newcomer" issue.

The application period was a month long and I thought that would be plenty of time to make some decent contributions -- wrong. I severely underestimated how much I had to learn, including:

Learning about over a dozen organizations that were offering internships in the program.

I couldn't work with all of them, so I had to read about them and try to narrow it down to a handful of possibilities using the criteria I discussed previously. Usually, I could figure out the overall mission of an organization, but if I went a level deeper and looked at the different components of their software project, I got lost in technical jargon.
Installing the Linux operating system

I was a life-long Windows user and had wanted to try Linux, but had never gotten around to it in the last 5+ years. Fortunately, my laptop is a Thinkpad T400 with an Ultrabay slot,³ which made it especially easy to install Linux distros. Still, it took 2-3 days to figure out how to partition my hard drive and install the Linux distros (Ubuntu and Debian) with the basic packages.
Learning how to "install" software in general

By install, I mean in the general sense -- not just installing things like Ubuntu and Devstack (the development version of OpenStack), but also situations like when you build a test environment and have to install lots of dependencies. Outside of writing and thinking about code, most of my technical problems had to do with missing dependencies.⁴ I learned to fix these issues by searching for the names of packages on the internet and in the bug trackers.
Learning specific version control systems and tools

FOSS organization use a variety of version control systems and tools, especially if they've been around for a while. I encountered Subversion, Git, and Gerrit (a code review tool built on top of Git). Fortunately, I had a little experience using Subversion and Mercurial, but learning Git on the fly is not ideal⁵ especially if something "breaks." One organization I worked with used Git, so I thought it was sufficient to learn Git, but I didn't know they also used GitHub to submit patches via pull request. This caused me a lot of trouble because I had cloned their source repository directly rather than from a forked repo on GitHub. When I tried to submit my patch, my Git push failed because my forked repo didn't know that my local repo existed. I had no idea how to fix this, and Christopher Lonnen, a Mozilla developer, spent an hour on both a Friday and a Saturday helping me -- I'm glad he was so generous with his time!
Learning the code review process

Submitting patches to fix typos or add documentation was straightforward, but if it was more complicated, I was lost. It's not trivial to figure out how to fit your work into a large code base. There were things I didn't know, for instance, that I needed to write or revise a unit test to go along with a feature/fix, or that I could submit incremental patches containing partial work. I was unfamiliar with code style guidelines like PEP8 and submitted code with trailing whitespaces.
Learning how to work in virtual environments

When it comes to helping newcomers get onboard a FOSS project, the focus seems to be on teaching them about version control and bug tracking systems which is great, but virtual environments are neglected. I found myself using virtualization tools like virtualenv and VirtualBox without having any knowledge of why virtualization was needed or how it works. It was difficult to find online tutorials on virtualization.
Getting used to IRC and FOSS culture.

Up to this point, I had chatted privately on clients like ICQ/Pidgin/Gchat, but I had never been in a public chatroom before. It took me some time to pick up chat slang like "np", "yw", "o/" and become accustomed to sitting on IRC channels all day so I could chase people down. But more than that, I wasn't sure how to conduct myself on a public channel. It felt unnatural and intimidating for the first few days, and that was even with nice people noticing me on the channel and welcoming me! Some anecdotes:
- I had great experiences asking for Linux install help on #ubuntu and #debian. Assistance came immediately, so I'm very grateful to those communities.
- Also, a huge thank you to Sumana Harihareswara of the Wikimedia Foundation who tagged along as my buddy to a (non Wikimedia) software development channel. I was nervous about "talking" to a bunch of strangers on a public channel and her presence as "wing woman" alleviated my fears.
- Sadly, my first day on IRC, I had an interaction with a male that I would classify as "marginal stalking." I informed the operators of the channel, and they dealt with the matter swiftly. It bothered me a little, but it would have been much worse if no one had been there to take care of me.

Another obstacle was finding things to work on. For the sake of the application process, some organizations would come up with "easy" bugs for the applicants, but I'm not sure if this is the norm. I recall that when I tried to work with OpenStack during the application period, they had no idea where to find easy bugs, at first.

I had (what seemed to me) a great background for getting into open source -- strong technical experience from studying science, lots of software user experience (FOSS and proprietary), strong interest in coding, native English speaker, familiarity with nerdy pop culture, the right hardware (Thinkpad).

Even then I had to get over a major hump in order for me to make my first contributions.

Some of this might have been easier if I had more experience with software development and Linux. However, I put off learning Linux for over 5 years because I never had a real reason to use it! Working in FOSS was my driving impetus to learn software development and Linux. So that is the "chicken and egg" problem, at least for me personally.

Working on OpenStack

At the end of the application period, I picked OpenStack as my first choice organization.⁶ Happily, I was selected, and shortly afterwards, began working on OpenStack.

It's true that I made my first FOSS contributions while applying to Outreach Program for Women, but those were simple patches like fixing typos and adding documentation. Being an OpenStack intern provided me with my first experience in working on substantial patches that needed to work with a large codebase and that required significant code review.

I found out later that OpenStack is not an easy project for newcomers to get into. Recently, while I was on IRC, I had an interesting discussion with someone from India. The person told me that OpenStack has a "tough" reputation. They tried to install Devstack unsuccessfully, and their friends had the same experience.

I was lucky to have the help of Anita Kuno, a recent OpenStack intern alumnae, who put heroic, generous effort into getting me started. She showed me how to install Devstack step-by-step and how to use Gerrit. When needed, she talked to OpenStack people on my behalf. After my very first contribution to OpenStack's Gerrit system, I had a strange, private interaction on IRC with someone who saw my patch. I told Anita about it, and she dealt with the person immediately.

Here at OpenStack, I have my team of supporters. Besides Anita, I have Julien Danjou (my mentor), Julie Pichon (former Outreach Program for Women mentor), and Anne Gentle (OpenStack's Outreach Program for Women coordinator) as well as past OpenStack interns Victoria Martínez de la Cruz and Laura Alves.

It goes without saying that having a good mentor is essential. I'm always able to get in touch with Julien, no matter how busy he is as Ceilometer Project Technical Lead. During the Outreach Program for Women application process, I interacted with some mentors who seemed "too busy" and found it frustrating, so I truly appreciate Julien's effort to be available. It's also very helpful to have the "team" of supporters I just mentioned. When someone is busy, I can try someone else.⁷ All the people on my "team" work on different things and have varied backgrounds, which means they have different perspectives. This is also extremely educational and useful for me.

Personally, I've found that good mentoring is not purely based on technical knowledge and experience. While those things are important (and essential to have in at least one mentor, in my case, Julien), I think good judgment and thoughtfulness are equally relevant. People who have less experience can mentor, too, if they are thoughtful. They can help out with the things they do know about and refer me to others when they feel like a matter is outside their domain of experience. It's very useful to have a "cultural ambassador" who can show you around the FOSS community and be a role model for how to behave.

I'm not saying that technical knowledge isn't necessary at all in mentoring; I'm merely trying to highlight some other factors which may not be as obvious.

Beyond mentoring, here are some things I experienced/learned while being a new contributor to OpenStack:

I wasn't sure how much or how little I could bother people on the development channels. Hypothetically, at what point would I bother people to look at my patch? (In reality, I've always gotten prompt reviews.)
I work on OpenStack in a virtual machine,⁸ which gives me peace of mind. If the development environment becomes irreversibly screwed up, I can always trash my current virtual machine and start a fresh one. Then after installing Devstack, I can resume work on my current patch using the command git review -d patch####.⁹ The point is that I shouldn't fear making mistakes in my local development environment, if it's on a virtual machine.
If people know I'm new to software development and the project, it's okay to submit an incremental patch. What I mean is that if I'm not sure what I'm doing and the patch can be naturally partitioned, I can submit the patch part by part. For example, if I'm working on a feature in multiple back-end drivers, I could divide the patch into 4 parts: 1) test for the drivers, 2-4) implementation for each of 3 drivers. People can comment as I submit each part, so that I know I'm on the right track.
I can mark patches on Gerrit as "drafts" or use the "work in progress" status if my patch is not ready for a major review.
It's important to know who to go to when there is a significant issue, especially if it's relevant to the entire community. If I have technical issue, I can go to my mentor Julien Danjou. If I have a non-technical issue, I can go to Anne Gentle (OpenStack intern coordinator) or one of the Outreach Program for Women coordinators, Marina Zhurakhinskaya and Karen Sandler. I hope everyone has someone they can go to, in their own FOSS community.

The one important thing I learned about contributing to open source

This might sound clichéd, but I think open source is very much about the people. You can learn a lot from internet tutorials, but there are always unforeseen obstacles and you need people to guide you through them.

Thanks to many many kind, supportive individuals I was able to

install Linux,
repair my Git history,
survive the Devstack gauntlet,
fend off weirdos,

and most importantly, feel welcome as a bona fide member of the open source community.

For an interesting take on how to assess open source projects, see Mel Chua's blog post "5 minutes of improvisation". ↩
I've written previously about the Outreach Program for Women -- explaining what the program is about and how I learned of its existence. ↩
Thinkpads are among the best supported machines in the Linux world, and the secondary slot allowed me to install a second hard drive that I could run Linux on, without any risk to my primary hard drive with Windows on it. I also had my RAM maxed out to 8 GB, which was very helpful later on, when I ran Devstack in a virtual machine. ↩
I recently wrote a OpenStack-related post about how to install Devstack and Ceilometer. In the troubleshooting sections, I mention lots of bugs related to missing dependencies, all of which I encountered while trying to install software or run tests. ↩
I did go through a very short Git tutorial, but in my opinion, it wasn't sufficient. The application process put me under time pressure, so I wasn't able to do a more thorough tutorial. ↩
You can read my "Why Openstack" post to find out why I picked them! ↩
I speculate this is the reason that Google Summer of Code requires multiple mentors for each intern. Google Summer of Code is an FOSS internship program similar to Outreach Program for Women, but it focuses on coding and is restricted to students. ↩
My post about installing Devstack and Ceilometer includes a section about using Vagrant/VirtualBox virtual machines. ↩
This command automatically downloads the patch and creates a branch for the patch in your local Git repository. ↩
The image I used for this post is a screenshot of my very first OpenStack contribution. ↩

How to install Devstack and Ceilometer on a Vagrant virtual machine

2013-07-30T00:00:00-04:00

DISCLAIMER: THIS INFORMATION IS EXTREMELY OUT OF DATE.

Ceilometer is a fairly new component of OpenStack, so I thought it would be helpful to document how I install Ceilometer and DevStack. Ceilometer monitors and gathers data on other components in OpenStack. For example, one could use Ceilometer to measure CPU usage in order to bill customers.

In this post, I'll go through how I install Devstack and Ceilometer on a Vagrant virtual machine running 64-bit Ubuntu 12.04 LTS. Your host machine should have enough memory, so that it can run a virtual machine with 2 GB RAM.

Before getting into details, some disclaimers and explanations.

I assume that if you want to install Devstack, it's because you're interested in being a contributor to OpenStack development or because you just want to play with the code. Devstack is a shell script that builds an OpenStack development environment for people developing OpenStack; it's not meant for production.

I install Devstack and Ceilometer on a virtual machine. Why use a virtual machine? It's the recommended way to install Devstack. Cloud infrastructure software is complex. Devstack installs lots and lots of dependencies, and you probably don't want all those dependencies to be installed on your everyday environment. Moreover, if Devstack or Ceilometer breaks (which happens pretty regularly when you develop), it's a lot easier to start over from scratch if you're working in a virtual machine. You can simply destroy the old virtual machine and build a new one, without any effect on your everyday environment.

I'm using Vagrant because it was recommended to me, but there are plenty of other ways to run virtual machines. You are welcome to use your own method, but this post will only explain how to setup virtual machines using Vagrant.

Finally, note the date of this post. Devstack and Ceilometer are rapidly being developed, and this information could become deprecated very quickly.

Install VirtualBox and Vagrant
Configure Vagrant
Install Devstack and Ceilometer
Check that Devstack is working
Configure Devstack for Ceilometer
Check that Ceilometer is collecting data
Run Ceilometer tests
References

1. Install VirtualBox and Vagrant

Vagrant is software for building and distributing virtual development environments. It works with many virtualization software packages. By default, Vagrant uses VirtualBox for its virtualization software package.

Install VirtualBox

First, install VirtualBox. Follow the instructions on the VirtualBox Linux downloads page.

I'm running 64-bit Ubuntu 12.04 and here's what I did. First, I added the following line to my etc/apt/sources.list file:

deb http://download.virtualbox.org/virtualbox/debian precise contrib

Then I downloaded and added the Oracle public key using

$ wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -

I checked that I had the correct key by examining its fingerprint:

$ apt-key finger | grep VirtualBox -3

Finally, I installed VirtualBox by running the commands:

$ sudo apt-get update
$ sudo apt-get install virtualbox-4.2

Install Vagrant

Download the latest version at the Vagrant downloads page, as of this writing, v1.2.7. (The version at the Ubuntu repository is a bit behind the latest.) Then install it using your package manager. On 64-bit Ubuntu 12.04, I ran these commands to install Vagrant:

$ wget http://files.vagrantup.com/packages/7ec0ee1d00a916f80b109a298bab08e391945243/vagrant_1.2.7_x86_64.deb
$ sudo dpkg -i vagrant_1.2.7_x86_64.deb

2. Configure Vagrant

Next, create a directory where you will manage your Vagrant virtual machine (VM), for example,

$ mkdir ~/vagrant_devstack

Install Vagrant box

Now you need to pick a "box." Vagrant uses "boxes" as the base image for creating a virtual environment. A box is based on a specific operating system possibly with additional software and configurations. Vagrant has official bare-bones Ubuntu boxes, and there are also many other unofficial boxes at the community page located at www.vagrantbox.es.

I recommend sticking to the official boxes. Devstack officially supports Ubuntu 12.04, so install the 64-bit Ubuntu 12.04 box for VirtualBox:

$ vagrant box add precise64 http://files.vagrantup.com/precise64.box

(Yes, it's confusing that both my host machine and my Vagrant VM are running 64-bit Ubuntu 12.04 ! Your configuration may differ.)

A 64-bit OS is preferred, so that you can assign more RAM to your VM. Note that since we are running a 64-bit guest machine on VirtualBox, you need to make sure that hardware virtualization is enabled on your host machine. This may require you to access your machine's BIOS settings.

Create and configure Vagrantfile

Create a Vagrantfile using the commands

$ cd ~/vagrant_devstack
$ vagrant init precise64

and you should see the output

A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

This creates a Vagrantfile in the directory where your Vagrant VM will be managed. The Vagrantfile is the configuration file for your Vagrant VM.

We are going to setup port forwarding and increase the RAM by altering the Vagrantfile. Open the Vagrantfile in your favorite text editor and see if there are any lines matching

# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# config.vm.network :forwarded_port, guest: 80, host: 8080

If you find the matching lines, uncomment the last line so that you have

config.vm.network :forwarded_port, guest: 80, host: 8080

If you don't have the matching lines, then manually copy and paste the above line to the Vagrantfile.

We are using port 8080, but you can use another port, as long as it isn't being used by another process. We will need port forwarding later to check that Devstack is working properly.

The recommended amount of RAM for Devstack is at least 2 GB. However, the default RAM in a Vagrant VM is 360 MB, so we need to change this.

Add the following lines to the Vagrantfile:

config.vm.provider :virtualbox do |vb|
  vb.customize ["modifyvm", :id, "--memory", "2048"]
end

My laptop I work on has 8 GB RAM, and it works fine running a 2 GB Vagrant/VirtualBox VM.

Here is a sample Vagrantfile that configures both port forwarding and 2 GB RAM.

3. Install Devstack and Ceilometer

Boot up the Vagrant VM

We are ready to install Devstack and Ceilometer. First, create the virtual environment and boot the virtual machine by running the commands

$ cd ~/vagrant_devstack
$ vagrant up

and you should see the output

Bringing machine 'default' up with 'virtualbox' provider...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] -- 80 => 8080 (adapter 1)
[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
[default] Mounting shared folders...
[default] -- /vagrant

Now you can SSH into the machine via

$ vagrant ssh

and you should see the output

Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
Welcome to your Vagrant-built virtual machine.
Last login: Fri Sep 14 06:23:18 2012 from 10.0.2.2
vagrant@precise64:~$

Check that your virtual machine has 2 GB RAM:

vagrant@precise64:~$ grep MemTotal /proc/meminfo
MemTotal:        2051540 kB

If you see a different number, check your Vagrantfile.

Install Devstack and Ceilometer

Now that you are logged into your Vagrant VM, the first thing to do is download and update the package lists from the Ubuntu repositories:

vagrant@precise64:~$ sudo apt-get update

Next, install git and your favorite terminal text editor, for example,

vagrant@precise64:~$ sudo apt-get install git vim-gtk

Clone the Devstack GitHub repository:

vagrant@precise64:~$ cd ~
vagrant@precise64:~$ git clone git://github.com/openstack-dev/devstack.git

then copy the sample localrc to the top-level of the devstack repository:

vagrant@precise64:~$ cd ~/devstack
vagrant@precise64:~$ cp samples/localrc .

This file localrc allows you to configure user-defined variables in the stack.sh script which installs Devstack.

(If you previously installed Devstack using the stack.sh script, you can start the installation process for Ceilometer here. The only difference is that you won't see the prompt for an admin password.)

Ceilometer is not enabled by default, so add the following to the end of your localrc file:

# Enable the ceilometer services
enable_service ceilometer-acompute,ceilometer-acentral,ceilometer-collector,ceilometer-api

Now you are ready to install Devstack and Ceilometer! This is a CPU intensive process, so don't run any heavy programs in the background.

Run the script to do the installation:

vagrant@precise64:~$ ./stack.sh

and you should see the output

Using mysql database backend

################################################################################
ENTER A SERVICE_TOKEN TO USE FOR THE SERVICE ADMIN TOKEN.
################################################################################
This value will be written to your localrc file so you don't have to enter it 
again.  Use only alphanumeric characters.
If you leave this blank, a random default value will be used.
Enter a password now:

Input a password as directed and sit back as everything is installed for you. It took about 30 minutes to install on my own VM.

The Ceilometer GitHub repository will be cloned to your /opt/stack/ceilometer directory.

I already installed Devstack, but not Ceilometer

If for whatever reason you already installed Devstack but have not yet installed Ceilometer, no problem. You don't have to create a new VM. Skip down to the part in the instructions where it says you should begin if you previously installed Devstack.

Troubleshooting

If you have any problem whatsoever while running ./stack.sh, it is a good idea to check the logs in the directory /opt/stack/logs. The screen output from running ./stack.sh is saved there and the logs have names like stack.sh.log.<date>. The most recent log is saved in the file stack.sh.log.

If you don't see any logs there, then check your Devstack configuration in the localrc file. There should be a line that sets the LOGFILE variable, for example,
```
LOGFILE=$DEST/logs/stack.sh.log
```
For more info, see the "stack.sh logging" section in the Devstack stack.sh documentation.
If you see an error immediately after running ./stack.sh like
```
ImportError: No module named netaddr
```
then install the dependency python-netaddr:
```
vagrant@precise64:~$ sudo apt-get install python-netaddr
```
This is a bug I've encountered, but it should be fixed now.
If you see many errors of this type while running ./stack.sh:
```
ERROR:root:Could not find any typelib for GnomeKeyring
```
then install the dependency gir1.2-gnomekeyring-1.0:
```
vagrant@precise64:~$ sudo apt-get install gir1.2-gnomekeyring-1.0
```
This is a known bug.
If stack.sh complains that there is no mongo client, then install MongoDB packages that are version 2.2 or higher (this is the current requirement for Ceilometer). If there are Ubuntu MongoDB packages with version 2.2 or higher for your operating system, you can install them with
```
vagrant@precise64:~$ sudo apt-get install mongodb-clients mongodb-server
```
If you are using Ubuntu 12.04 (like we are in this guide), as of this writing,the MongoDB packages for Ubuntu 12.04 are too old (< 2.2). In that case, install the 10gen version mongodb-10gen by following the instructions here.

4. Check that Devstack is working

On your host machine (not the VM), open up your browser and point it to the forwarded port. If it was 8080, you would use http://localhost:8080/. You should see the OpenStack dashboard page with a login form:

To login, provide the username admin and the corresponding password which can be found in the ADMIN_PASSWORD variable of your localrc file.

Go to the project page at http://localhost:8080/project/ and you should be able to see quotas like Instances Used 0 of 10, VCPUs Used 0 of 20, RAM Used 0 MB of 51,200 MB, etc.

If you go to the system info page at http://localhost:8080/admin/info/, you should be able to see all the enabled services:

including nova, glance, ceilometer, etc.

5. Configure Devstack for Ceilometer

We want to enable auditing events for Nova and Cinder, so that Ceilometer can collect data from them. As stated in the Ceilometer documentation, do the following. First, check to see if the Nova configuration file /etc/nova/nova.conf contains a line:

instance_usage_audit = True

If the line does not exist, add it to the file.

Second, add this line to the end of the Cinder configuration file /etc/cinder/cinder.conf:

notification_driver=cinder.openstack.common.notifier.rpc_notifier

if it doesn't already exist.

6. Check that Ceilometer is collecting data

We can check that Ceilometer is collecting data by starting up the MongoDB shell, connecting to the Ceilometer database, and examining the data.

First, start up the MongoDB shell and connect to the Ceilometer database by running

vagrant@precise64:~$ mongo ceilometer

which should produce the output

MongoDB shell version: 2.0.4
connecting to: ceilometer
>

and leave you at the prompt.

Ask for the number of samples that have been collected:

> db.meter.stats().count

You should get a number greater than 0; I got 19 samples. As time goes on, the number will increase.

The command db.meter.findOne() should return the information for one sample, for example,

> db.meter.findOne()
{
    "_id" : ObjectId("51f85d6a56c02c63d2d503b3"),
    "counter_name" : "image.upload",
    "user_id" : null,
    "message_signature" : "3986da6d1ac0ef522f11c52dbf76119d9f93c79204fa3ca373017c78acb751a7",
    "timestamp" : ISODate("2013-07-31T00:42:17.639Z"),
    "resource_id" : "6fd80822-b6e5-4a8c-8e37-caec037ff19b",
    "resource_metadata" : {
        "status" : "saving",
        "created_at" : "2013-07-31T00:42:17",
        "event_type" : "image.upload",
        "deleted" : false,
        "container_format" : "aki",
        "min_ram" : 0,
        "disk_format" : "aki",
        "updated_at" : "2013-07-31T00:42:17",
        "name" : "cirros-0.3.1-x86_64-uec-kernel",
        "min_disk" : 0,
        "host" : "precise64",
        "protected" : false,
        "id" : "6fd80822-b6e5-4a8c-8e37-caec037ff19b",
        "checksum" : "c352f4e7121c6eae958bc1570324f17e",
        "owner" : "728bca2e0c5147e6952427de5a21f90e",
        "is_public" : true,
        "deleted_at" : null,
        "properties" : {

        },
        "size" : 4955792
    },
    "source" : "openstack",
    "counter_unit" : "image",
    "counter_volume" : 1,
    "project_id" : "728bca2e0c5147e6952427de5a21f90e",
    "message_id" : "0d41abf8-f97a-11e2-b536-080027880ca6",
    "counter_type" : "delta"
 }

Running the command show collections lists all the other MongoDB collections you can look at, for example,

> show collections
meter
project
resource
system.indexes
user

The Ceilometer documentation glossary defines some of these terms "meter", "project", "resource", "user".

You could list all the defined indexes using

> db.system.indexes.find()

which on my VM gave the output

{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "ceilometer.resource", "name" : "_id_" }
{ "v" : 1, "key" : { "user_id" : 1, "source" : 1 }, "ns" : "ceilometer.resource", "name" : "resource_idx" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "ceilometer.meter", "name" : "_id_" }
{ "v" : 1, "key" : { "resource_id" : 1, "user_id" : 1, "counter_name" : 1, "timestamp" : 1, "source" : 1 }, "ns" : "ceilometer.meter", "name" : "meter_idx" }
{ "v" : 1, "key" : { "timestamp" : -1 }, "ns" : "ceilometer.meter", "name" : "timestamp_idx" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "ceilometer.user", "name" : "_id_" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "ceilometer.project", "name" : "_id_" }

These indexes speed up searching and sorting on MongoDB data.

Type exit to leave the MongoDB shell and return to bash.

For more information about MongoDB shell commands, see the official MongoDB documentation.

7. Run Ceilometer tests

The Ceilometer documentation explains how to setup and run unit tests. First, install the package tox:

vagrant@precise64:~$ sudo pip install tox

Then install the test dependencies:

vagrant@precise64:~$ sudo pip install -r /opt/stack/ceilometer/test-requirements.txt

This will take at least a few minutes to run. If the dependencies were installed correctly, you should get a message that looks similar to

Successfully installed discover mock swift netifaces sphinxcontrib-pecanwsme docutils oslo.sphinx testscenarios nova dnspython sphinxcontrib-httpdomain
Cleaning up...

Run the entire test suite in Python 2.7 and check for PEP8 style errors:

vagrant@precise64:~$ cd /opt/stack/ceilometer/
vagrant@precise64:/opt/stack/ceilometer$ tox -e py27,pep8

This will take a while. On my VM, the tests took 10 - 15 minutes to run.

If everything runs correctly, you will get a message similar to

  py27: commands succeeded
  pep8: commands succeeded
  congratulations :)

The OpenStack dashboard is working, Ceilometer is collecting samples, and the unit tests pass. Congratulations! You have a working Devstack and Ceilometer!

More ways to run tests

The Ceilometer unit tests are run using testr (manual here).

You can also run just the Python 2.7 tests

vagrant@precise64:~$ tox -e py27

or just the PEP8 tests

vagrant@precise64:~$ tox -e pep8

If you don't need to run the whole test suite, you can run a group of tests. To run the Python 2.7 tests in the tests/storage directory of the Ceilometer code, you would execute

vagrant@precise64:~$ tox -e py27 -- storage

If you want to run just a single test from tests/storage, you need to use the entire file path:

vagrant@precise64~$ tox -e py27 -- tests.storage.test_impl_mongodb.MeterTest.test_get_meters_by_metaquery

Here we're running the test_get_meters_by_metaquery test for the MongoDB storage driver.

Anything after -- is treated as an argument for testr. Each argument is treated as a regular expression. That leads to an interesting scenario.

When you are running tests in testr by path, be careful if your specified path is a substring of other paths. For example, if you run the test test_get_meters using the command

vagrant@precise64~$ tox -e py27 -- tests.storage.test_impl_mongodb.MeterTest.test_get_meters

the following tests will be executed -- test_get_meters, test_get_meters_by_user, test_get_meters_by_project, test_get_meters_by_metaquery, test_get_meters_by_empty metaquery -- because they all start with test_get_meters and are counted as matches by the regex filter.

Troubleshooting

If you see an error message
```
StorageBadVersion: Need at least MongoDB 2.2
```
then your MongoDB version is too old and you need to install MongoDB packages that are version 2.2 or higher (this is the current requirement for Ceilometer).

If there are Ubuntu MongoDB packages with version 2.2 or higher for your operating system, you can install them with
```
vagrant@precise64:~$ sudo apt-get install mongodb-clients mongodb-server
```
If you are using Ubuntu 12.04 (like we are in this guide), as of this writing,the MongoDB packages for Ubuntu 12.04 are too old (< 2.2). In that case, install the 10gen version mongodb-10gen by following the instructions here.
If you see the following error while installing the requirements:
```
No package 'libffi' found
```
then install the missing dependency with
```
vagrant@precise64:~$ sudo apt-get install libffi-dev
```
This is a known bug.

If you get the following error while installing the requirements:

RuntimeError: No package configuration found for: nspr

then install the missing dependencies with

vagrant@precise64:~$ sudo apt-get install pkg-config libnspr4-dev

If you get this error while installing the requirements that says

ERROR: /bin/sh: 1: xslt-config: not found

** make sure the development packages of libxml2 and libxslt are installed **

then install the missing dependencies with

vagrant@precise64:~$ sudo apt-get install libxml2-dev libxslt1-dev

If all else fails, try rebuilding the test environment with the -r flag:
```
vagrant@precise64:~$ tox -r -e py27
```
This can take anywhere between 15 minutes to an hour depending on the configuration.

8. References

I've mentioned some references to documentation along the way, here's a summarized list of them, plus a couple more.

Vagrant documentation

This is v2 of the documentation for Vagrant. However, I sometimes find it helpful to look at the deprecated documentation for v1. The v1 documentation sometimes contains details that are not present in the v2 documentation and yet are still applicable to the latest version of Vagrant. The Vagrant commands I use most often are vagrant up (boot up the VM), vagrant ssh (ssh into the VM), vagrant init (create a Vagrantfile in preparation for making a new VM), and vagrant destroy (permanently destroy the data I've accumulated while working in the VM).
VirtualBox documentation
Devstack documentation

Also, check out the code annotations for the Devstack stack.sh script
Ceilometer documentation
MongoDB documentation
testr documentation

Why OpenStack

2013-07-17T00:00:00-04:00

Recently, Anne Gentle, the OpenStack Outreach Program for Women coordinator, told me it would be interesting to know what made OpenStack appealing to me personally. I'll try to answer that question here.¹

A major constraint is simply that I applied to the Outreach Program for Women and there are a limited number of organizations to choose from. Still, there are 19 organizations and that is a quite a few to choose from! Why OpenStack?

I was interested in working on a project with great impact and that had some relevance to my science background.

Not to belittle other organizations, but I think it's easy to argue that OpenStack is an extremely important project with broad interest to many groups. I counted 61 official corporate sponsors on this page. There are many impressive names on the list ranging from FOSS (Ubuntu, RedHat, Rackspace) to bluechip tech (IBM, HP, Intel) to communications (AT&T, Comcast) to storage (Seagate, Hitachi) and many others that I'm omitting. Large is not necessarily equal to significance, but this infographic shows how rapidly OpenStack has grown in 3 short years.

Regarding my interest in science, both Argonne National Laboratory and San Diego Supercomputer Center have been using OpenStack, and they are major national science labs in the USA. My sister's PhD advisor in linguistics, when he learned that I was working on OpenStack, remarked that he was glad because "OpenStack is an important project." As someone who's enthused about numerics and computational science, I'm excited about the prospect of scientists crunching large-scale calculations about the origins of the universe or protein folding on a FOSS cloud running OpenStack.

On a personal level, I have observed how deeply embedded cloud computing has become in our everyday lives. I was thinking about how much I use Gmail. I don't know how I would replace it with a FOSS alternative; it's hard to beat the security and widespread accessibility of Gmail. I'm not aware of any FOSS email tool that can be accessed and synced reliably on desktop browser, POP email client, tablet, and phone. I also stream music, host GBs of photos, and share social media -- all from the (corporate) cloud.

Knowing that OpenStack is actively being developed gives me the piece of mind, that someday technologically minded people, including myself, could do all those things from our own personal clouds or start their own smaller "indie" versions of Gmail, Facebook, Twitter, etc. In particular, for government agencies and scientific labs, it makes more sense (at least to me) to use FOSS that is more politically neutral, transparent, and accessible to the general public. If government funded institutions use corporate clouds, they might get locked into a particular stack of software. From my experience with the National Day of Civic Hacking, I saw how crucial it is for the government to use FOSS tools, so that vigilant citizens can examine the data for themselves.

Moreover, I always want to be expanding my knowledge. Cloud computing and software/infrastructure as a service are topics for which I have very little technical understanding. I'd also like to learn enough that I explain what cloud computing is to non-technical friends and family; in general, I'd like to educate the public about the software they rely on!

There are a couple more, less lofty reasons that I became an OpenStack contributor. I'm interested in getting a job in the software industry, but don't have any experience working on large scale, multiple user projects. Working on a project like OpenStack gives me credibility because of its scale, complexity, significance, and strong reputation.

Another reason I liked OpenStack was that it used Python, a language I know and enjoy using. It's also nice that Python is quite popular in general and extensively used in the scientific community. I didn't want to work in a language I disliked or that was too obscure.

As I finish writing this post, I realize that I would have wanted to contribute to OpenStack even if FOSS internship programs were unavailable. There are so many good reasons to contribute to the project, including the fact that everyone in the community has been so welcoming and supportive. I just wish they had publicized themselves more at universities, so I could have known about the organization sooner.

Here's to a long life for OpenStack!

Since I'm new to open source software, cloud computing, and OpenStack, my answer might be a bit naive or technically incorrect. My apologies in advance. ↩

How I setup Pelican

2013-07-01T00:00:00-04:00

DISCLAIMER: This information may be out-of-date. This post was written using Pelican version 3.2 and the themes and plugins in July 2013. However, last I checked (August 2015), almost everything still works.

I thought it would be useful to go into greater technical detail about how I went about setting up Pelican.¹

My machine is running Ubuntu 12.04 LTS.

Summary:

Install Pelican and its dependencies in a virtual environment.
Run pelican-quick-start
Fork the Pelican themes and Pelican plugins on GitHub and clone the forks onto my local machine.
Modify Pelican themes to my taste. I modified the bootstrap theme.
Make additional directories inside content directory.
Modify settings in pelicanconf.py and publishconf.py
Setup Google Analytics/Piwik and Disqus.

Demonstration

If you'd like to see some of Pelican's features in action (Markdown, formatting, $\LaTeX$, etc.), see my earlier post on demonstrating Pelican.

Installing Pelican and dependencies

To install Pelican and other Python package dependencies in a virtual environment, I followed the instructions in the official Pelican documentation.

$ virtualenv ~/virtualenvs/pelican
$ cd ~/virtualenvs/pelican
$ . bin/activate
$ pip install pelican
$ pip install Markdown

Don't use sudo pip when installing these Python packages in a virtual environment, otherwise you'll end up installing them globally which defeats the purpose of using virtual environments.

I wanted to use Markdown to write my blog posts, so I needed to install the Python Markdown package in the last line. By default, Pelican uses a flavor of Markdown that mostly follows PHP Markdown Extra. I especially like using the footnotes extension.

After you're done working with Pelican, you can leave the virtual environment with

$ deactivate

Running pelican-quickstart

Here are the configuration options I used for pelican-quickstart. The details for SSH upload have been anonymized for security reasons.

$ pelican-quickstart
Welcome to pelican-quickstart v3.2.2.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.

> Where do you want to create your new web site? [.] blog
> What will be the title of this web site? terriyu@/media/blog 
> Who will be the author of this web site? Terri Yu 
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) Y
> What is your URL prefix? (see above example; no trailing slash) http://terriyu.info
> Do you want to enable article pagination? (Y/n) Y
> How many articles per page do you want? [10] 
> Do you want to generate a Makefile to easily manage your website? (Y/n) Y
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) Y
> Do you want to upload your website using FTP? (y/N) N
> Do you want to upload your website using SSH? (y/N) Y
> What is the hostname of your SSH server? [localhost] ip.add.ress
> What is the port of your SSH server? [22] 
> What is your username on that server? [root] username 
> Where do you want to put your web site on that server? [/var/www] /somedirectory 
> Do you want to upload your website using Dropbox? (y/N) N
> Do you want to upload your website using S3? (y/N) N

I enabled article pagination so that I didn't end up with all the posts on a single page. The Makefile is incredibly useful, so I enabled that. It allows you to compile the website with make html or make devserver. Usually, when I want to test my website, I run make clean (to delete any previously generated files), followed by make devserver to run a fake server on my local machine. I find that make devserver is a better simulation of the website than make html. I'm not sure what the auto-reload & simpleHTTP script is, but I enabled it just in case.

When I'm ready to upload my content to the web host, there are a variety of options. I don't enable FTP upload because it's not very secure. I try to stick with some transfer method that uses SSH. You can use make ssh_upload which works fine, but I prefer make rsync_upload which uploads using rsync and SSH. Unlike make ssh_upload which uses scp and uploads everything, the rsync command only sends incremental changes, which makes your data transfer much more efficient.

All the make commands have to be run in the same directory as the Makefile.

Fork and clone the GitHub repos for pelican-themes and pelican-plugins

I wanted to modify the official Pelican themes and Pelican plugins, and have them under version control, so I performed the standard procedure of forking the GitHub repos and cloning the forks. My procedure was very similar to the GitHub help page except I cloned using the SSH address rather than the https address.

Modify Pelican theme to taste

I am using the bootstrap theme. I modified it in the following way:

Modified static/bootstrap.min.css
- Increased font size and line height
- Changed sizes and alignment of top bar, sidebar, and content
- Eliminated line in page header
Modified bootstrap/static/local.css
- Changed icons to use Font Awesome's set
- Changed alignment of icons
Modified static/pygments.css to use the perldoc colors by generating a CSS stylesheet
Added code to templates/article.html for neighboring links. This means that if you are on a post, you will see a link for the previous post and a link for the next post. You need to enable the neighbors plugin for this to work.
Modified templates/base.html
- Added code for including $\LaTeX$. For this to work, you need to enable the latex plugin.
- Added code for including Piwik content
- Altered sidebar links
Modified templates/index.html to change look of the more... link in the summaries, got rid of the permalink line, and added the date before each post title
Modified templates/metadata.html to use the variable article.locale_date instead of article.date so that it uses my DEFAULT_DATE_FORMAT variable in pelicanconf.py
Added templates/piwik.html (copied from bootstrap2 theme) for Piwik support
Modified templates/tags.html to sort tags in alphabetical order

To sort tags in alphabetical order, use

{% for tag, articles in tags|sort %}

Make additional directories inside content directory

I made directories 2013, images, and pages in the content directory. This is the way I organized my content. Posts in 2013 go into 2013, photos go into images, and special non-chronological content goes into pages. I plan to make a new directory for each year and it would contain the posts for that particular year. The only directory that has to have a specific name is pages. Pelican treats this name as special. My About page is contained inside the directory pages.

For more information about the file hierarchy, see the Pelican documentation.

Modify settings in pelicanconf.py and publishconf.py

This is my pelicanconf.py file.

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

AUTHOR = u'Terri Yu'
SITENAME = u'terriyu@/media/blog'
SITEURL = ''

TIMEZONE = 'America/New_York'

DEFAULT_LANG = u'en'

# Feed generation is usually not desired when developing
FEED_ALL_ATOM = None
CATEGORY_FEED_ATOM = None
TRANSLATION_FEED_ATOM = None

# Feeds 
FEEDS =  (('All posts', 'feeds/all.atom.xml'),
          ('Category', 'feeds/category'),
          ('OPW', 'feeds/tag/opw.atom.xml'),)

# Social widget
SOCIAL = (('Github', 'https://github.com/terriyu'),
          ('Twitter', 'https://twitter.com/terrimyu'),
          ('Linkedin', 'https://linkedin.com/in/terriyu'),)

DEFAULT_PAGINATION = 10

# Uncomment following line if you want document-relative URLs when developing
RELATIVE_URLS = True

# Formatting for dates

DEFAULT_DATE_FORMAT = ('%a %d %B %Y')

# Formatting for urls

ARTICLE_URL = "posts/{date:%Y}/{date:%m}/{slug}/"
ARTICLE_SAVE_AS = "posts/{date:%Y}/{date:%m}/{slug}/index.html"

CATEGORY_URL = "category/{slug}"
CATEGORY_SAVE_AS = "category/{slug}/index.html"

TAG_URL = "tag/{slug}/"
TAG_SAVE_AS = "tag/{slug}/index.html"

# Generate yearly archive

YEAR_ARCHIVE_SAVE_AS = 'posts/{date:%Y}/index.html'

# Show most recent posts first

NEWEST_FIRST_ARCHIVES = False

# Specify theme

THEME = "/home/terri/pelican/pelican-themes/bootstrap"

# Plugins

PLUGIN_PATH = '/home/terri/pelican/pelican-plugins'
PLUGINS = ['latex', 'neighbors', 'summary']

# Only use LaTeX for selected articles

LATEX = 'article'

The configuration file is mostly self-explanatory. I uncommented the line for relative URLs. I also specified custom URLs for article, category, tag, and year archive, so that the URLs are more concise and don't have any .html extension. For example, instead of posts/2013/06/pelican-setup.html, I would have posts/2013/06/pelican-setup/. I added lines to show most recent posts first and to use the bootstrap theme. I also enabled the latex, neighbors, and summary plugins. The line LATEX = 'article' allows you to turn on $\LaTeX$ support by adding the line Latex: in the head of a post. I like using the summary plugin² because I can control exactly how much of each post is shown on the index page by using markers. Also, if I set the  marker before footnotes, then I avoid the bug where footnotes appear in the summary.

Below is my publishconf.py file with the Disqus and Google Analytics specifics anonymized.

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

# This file is only used if you use `make publish` or
# explicitly specify it as your config file.

import os
import sys
sys.path.append(os.curdir)
from pelicanconf import *

SITEURL = 'http://terriyu.info/blog'
RELATIVE_URLS = False

FEED_ALL_ATOM = 'feeds/all.atom.xml'
CATEGORY_FEED_ATOM = 'feeds/category/%s.atom.xml'
TAG_FEED_ATOM = 'feeds/tag/%s.atom.xml'

DELETE_OUTPUT_DIRECTORY = True

# Following items are often useful when publishing

DISQUS_SITENAME = 'disqus-sitename'
GOOGLE_ANALYTICS = 'google-analytics-id'

PIWIK_URL = 'url.where.piwik.install.is.located'
#PIWIK_SSL_URL = ''
PIWIK_SITE_ID = '1'

I mostly kept the default file, but added a tag feed.

Setup Google Analytics/Piwik and Disqus

Just go to the relevant website Google Analytics, Piwik, or Disqus and follow the directions on how to set it up. Then add the appropriate IDs to your publishconf.py file (see my example file from the last section). Piwik needs to be installed on a server. The major web hosts Bluehost and Dreamhost support one-click Piwik installs. Or, if you have a friend who is willing to share their Piwik install with you, that also works. The variable PIWIK_SSL_URL is for extra security and optional.

Top image taken from Pelican GitHub page ↩
Currently, there is an incompatibility between the summary plugin and the version 3.2 release of the Pelican package. The summary plugin in the pelican-plugins GitHub repository has been updated to be consistent with the development version of Pelican. Therefore, if you try to use the latest summary plugin from the GitHub repo with Pelican v3.2, you'll get an error related to the DEFAULT_CONFIG variable. To fix this, go into your cloned Git repo of pelican-plugins and manually change the code in summary/summary.py. Simply replace DEFAULT_CONFIG with _DEFAULT_CONFIG in the file. See this pull request for reference. ↩

Boston WiSE Software Carpentry boot camp

2013-06-27T00:00:00-04:00

I just finished up my stint as a helper at Software Carpentry's Women in Science and Engineering boot camp, also known as WiSE. The event was held at Microsoft's space in Cambridge Center near Boston from June 24 - 25.

Background

I was not an instructor. My post will be rather informal and detailed, from a helper's perspective. For a formal writeup from an instructor, see Aleksandra Pawlik's post for the Software Sustainability Institute. For a broad overview and perspective from someone new to Software Carpentry, see Kaitlin Thaney's post. Kaitlin is the new director of the new Mozilla Science Lab initiative. Since Software Carpentry now falls under Science Lab, Kaitlin is our boss... I think!

The month before, I had volunteered at the UMass Amherst boot camp. Since I am located reasonably close to Boston, I was asked to help out again at WiSE. The UMass boot camp was a low-key affair, but WiSE was a special, high-profile event for Software Carpentry. Any self-identified woman could register for the event, as long as she could make it to Boston and pay $20. This was unusual because up to this point, we ran boot camps at universities and research institutes with the participants mostly coming from that particular university/institute. Moreover, the WiSE boot camp was three times larger than normal and also had additional corporate sponsorship from J.P. Morgan, Microsoft, Intel, NumFOCUS, King Abdullah University of Science and Technology, and Python Software Foundation.

We had 7 instructors, 19 helpers, and approximately 120 participants -- all women! With so many people, we split the group into three rooms: novice, intermediate, and advanced. I was worried that we weren't going to have enough helpers¹, but at the last minute, J.P. Morgan sent a bunch of people, including some technologists from their Manhattan office. Thank you, J.P. Morgan! The instructors and helpers came from all over. Many of us flew in from across the country, some even internationally from Europe.

In addition, Amy Brown, Software Carpentry's admin, came from Toronto to keep us organized. She bought snacks, monitored communications, and generally helped to make sure things didn't fall apart.

Preparations

To prepare for the event, we held a conference call the week before, with all the instructors and helpers². The agenda was on an Etherpad and we added notes to it during the call. Tracy Teal, who I had worked with at the UMass Amherst boot camp, graciously volunteered her time to lead our merry band and did her best to organize us over the conference call and the mailing list.

The night before the boot camp, the instructional staff caught a little relaxation at the Cambridge Brewing Company. I recall trading stories about traveling, Star Trek, and college visits. It was a great idea to meetup before the event started, because the boot camp itself was absolutely hectic. It was astounding to me that almost all of the staff traveled from out-of-town. After our beer get-together, I was asked to show people to the T stop because I was the only person who had ever lived in the Boston area.

Boot camp

The boot camp ran from 9 am - 5 pm over two days, but we instructors/helpers arrived at Cambridge Center around 8 am, in order to prepare and help out with installation problems. Since there were so many people, we tried to identify ourselves as instructional staff by wearing Software Carpentry t-shirts, buttons, and stickers.

I was assigned to help in the advanced room and our instructors were Jessica McKellar, Aleksandra Pawlik, and Sheeri Cabal. Jessica taught shell and scientific Python, Aleksandra taught Git, basic Python, and testing, and Sheeri taught SQL. They are all software experts with impressive credentials. Jessica is an organizer for the Boston Python User's Group and on the Board of Directors for the Python Software Foundation. Sheeri organizes the Boston MySQL User Group. Aleksandra Pawlik works at the Software Sustainability Institute in England. The Software Sustainability Institute focuses on software in research settings and sounds very cool. I wish we had something like that here in the USA.

Just like at UMass Amherst, I spent the majority of my time manning the Etherpad -- taking notes and answering questions in the chat box. I was worried that people might be so "advanced" that they were bored, so I would sporadically write comments like "hey did you know that there are non-SQL databases" or "check out the history of unit testing" to entertain them. I don't know if this was helpful or not. In between Etherpadding, I checked to make sure our room was well-organized and supplied and helped people with their software installs. During quieter periods, I worked on promotion -- taking photos in each of the three rooms and sending out a few tweets.

Panel discussion

The evening of the first day, there was a reception and panel sponsored by J.P. Morgan. The panel consisted mostly of J.P. Morgan technologists, plus Jessica McKellar (Python expert / software engineer) and Wei Le (computer science faculty). Jessica and Wei were instructor and helper, respectively, in the advanced room. The panel was a discussion of women in software and how participants can get involved (see Aleksandra's post).

Unfortunately, I had to run to meet a friend for dinner, so I was only able to listen to the panel introductions. I did catch a funny remark by Prof. Le who said,

As a CS faculty, I look forward to work every day. Not true as a grad student.

As a former grad student, this made my day.

Problems, impressions, and anecdotes

As is usual in these boot camps, installation problems were a major issue. I heard reports that there were an unusually high number of install problems, partially because a number of people didn't bother installing anything before they came. Even though we provided a Python script to check installation and asked people to check their IPython version at the end of the first day, we still had quite a few problems in the advanced room. IPython notebook problems ate up the first hour of the second day. If you participate in a Software Carpentry boot camp or any software education event in the future, please please check your install as throughly as you can!!!

We were a bit disorganized in the advanced room, partially because it was hard to coordinate efforts with three instructors from different backgrounds. We forgot to set up the Etherpad and to ask for feedback on minute cards.³ However, once I realized this mistake, I informed the instructors and we rectified the situation quickly.

I got the impression that our advanced group was pretty bored during the shell lessons. A couple people wanted to know about pretty advanced shell stuff like sed and awk. The room perked up a bit when we went on to teaching Git. When we teach Git, I'm always worried that people think GitHub = Git or that GitHub is open source software (it's a commercial web service). Sure enough, someone asked if the Python software project used GitHub.⁴

A selection of my personal experiences:

I had a nice chat with a couple women about open source software, R, and Python. They were interested in doing data analysis. We all complained about how R makes it too easy to do irresponsible research -- you can run R libraries and obtain results that you don't understand.
I hung out with a participant after the first day and we talked about installing Python libraries on Mac OS X. I told her that we recommend all-in-one installers like Anaconda or Canopy to avoid issues during boot camps, but that in the long run, you want to be in charge of your software and should learn to install the libraries yourself with a tool like Pip.⁵ We also chatted a bit about machine learning.
Mixed feedback on virtual machines. On one hand, a participant gave us a huge thumbs up for the virtual machine (VM). She was really happy to have everything working out of the box and was running the VM on her Windows laptop. On the other hand, someone else was very unhappy that she couldn't get superuser access on the virtual machine. I tried to help her over the Etherpad and even asked the Software Carpentry instructors mailing list for the superuser username and password, but apparently she still couldn't get it to work.
A participant came up to me after the end of the second day and effusively gushed about how inspired she was by our teaching.

I learned from my time helping at UMass and tried to put together a list of resources and ideas for what people could do after the boot camp. At the UMass boot camp, people were really interested in follow up material, but the staff didn't have any formal list and we ended up pasting in URLs on the Etherpad. For a high-profile event like WiSE, I hoped that we could be more organized and that's why I attempted to come up with a collection of resources. You can see my list on the Software Carpentry blog.

Feedback

Looking at the minute cards, there were many different opinions on what was easy vs what was hard and what topics people wanted to cover. This shows how difficult it is to teach a large group, even when we separated participants by level in this particular boot camp! I think we could have done a better job presenting the goals of the boot camp. We should have had formal introductory and wrap up presentations -- talking about how we want to promote good software practices, reproducible research, and open science. The reality is that we simply can't meet everyone's exact needs, but we hope to give an overview of good software practices and to inspire people so that 1) they make their software a priority and 2) find a community of like-minded scholars.

Overall, we received lots of positive feedback from the participants and apparently enough interest to start a mailing list or Google Group. I hope the women from the boot camp continue to educate and empower themselves with software.

We normally desire at least a 2:1 helper:instructor ratio. ↩
At that point we didn't know who J.P. Morgan was sending, so they weren't on the conference call. ↩
For a "standard" Software Carpentry boot camp, we usually ask the participants to write 1-2 sentences about what was good and what was bad 1-2x each day. The feedback is written on index cards or stickies, and we tell them not to spend to long on it -- hence the term "minute cards." ↩
Not only does Python not use GitHub, the project doesn't even use Git as their version control system. They prefer Mercurial! ↩
This is my personal opinion and not necessarily Software Carpentry's official recommendation. ↩

UMass Amherst Software Carpentry boot camp

2013-06-18T00:00:00-04:00

A couple years ago, while I was in physics graduate school, I found out about an organization called Software Carpentry. My advisor sent me a Nature article about poor software practices in science and it quoted Greg Wilson, the founder of Software Carpentry. Around the same time, I also heard that Software Carpentry was running an online class to teach scientists software skills. I signed up and really enjoyed the class -- which covered Python, shell, version control, object oriented programming, regular expressions, testing, and databases.

I didn't know many people interested in software at grad school, so it was nice to be able to discuss these topics with the Software Carpentry instructors. I remember learning some nice Python tricks from Greg.

Since then, I've sporadically kept in touch with Greg. A couple months ago, he told me that he was running an onsite (offline) Software Carpentry bootcamp near me at UMass Amherst and asked me if I'd like to volunteer to help. I agreed and it turned out to be a fun and illuminating experience.

I had many thoughts about my experience at the boot camp and Greg Wilson, the founder of Software Carpentry, asked me to share my thoughts as a blog post.

You can see my roughly 2000 word blog post here on the Software Carpentry website.

First day of Outreach Program for Women

2013-06-17T00:00:00-04:00

Today is the first day of my internship in the Outreach Program for Women¹ (OPW). I'll be working on the Ceilometer component of the OpenStack cloud computing project under the guidance of my mentor, Julien Danjou, who is the project technical lead of Ceilometer. I'm also supported by two more mentors, Anita Kuno and Julie Pichon. Anne Gentle is the coordinator of OpenStack's OPW internship program, and Marina Zhurakhinskaya and Karen Sandler are the coordinators of the entire Outreach Program for Women. I'm grateful to them all.

I'll defer discussing my work on Ceilometer to another post, but I thought I'd first write about the internship program itself. The Outreach Program for Women started in 2010, as an effort to increase the number of women contributing to open source software. As stated on the program's main page:

This program has proved itself to be very effective in improving community diversity. Women attendance at GNOME's yearly conference, GUADEC, rose from 4% (6 women) to 17% (41 woman) in 3 years. A recent study showed that 50% of newcomers who joined the GNOME project in the last 3 years are women, while only 6% of newcomers to all other surveyed FOSS projects are women, with 15% being the next highest concentration.²

In each application round, various open source software organizations suggest projects, and each project has at least one mentor (who can male or female). The projects can be on any topic, as long as it is relevant to free open source software (FOSS). Many of the projects involve software, but there are also possibilities for working on marketing, design, and documentation.

From 2010 - 2011, the only organization involved was GNOME, but since then OPW has grown rapidly. In my application round, there were 19 organizations! The organizations were Debian, Electronic Frontier Foundation, GNOME, Joomla, KDE, Linux Kernel, MediaGoblin, Mozilla, NESCent, Open Technology Institute, OpenMRS, OpenStack, Perl, Tor, Twisted, Wikia, Wikimedia, WordPress, and Yocto Project.

OPW is modeled after Google Summer of Code (GSoC) but with a few differences.³ Like GSoC, OPW has an unusual application process. A significant part of the application is collaboration with mentor(s) and contributing to open source software. What does that mean? You can think of it as starting the internship early -- a pre-internship of sorts. The applicant looks through the list of organizations and their projects. She contacts the mentor(s) of the project(s) she is interested in and starts working with the mentor(s).⁴ The mentor(s) suggest some small contributions that the applicant can work on. A "contribution" can be fixing a typo in documentation or writing a test in the code, for example. By collaborating with the mentor and working on a contribution, the applicant gets a feel for the project and the mentor, and obtains experience in using the issue tracker/software/tools.

In my opinion, what's really cool about this process is that it allows the applicant and mentor to evaluate each other in true-to-life situations. It avoids the games you see in hiring processes, like asking logic puzzle questions or showing off credentials. Moreover, even if the applicant doesn't succeed in obtaining a spot, she has learned a lot and can apply again with a strong chance of succeeding the second time around.

Last winter 2012, I first heard about OPW on the Systers mailing list. Shortly afterwards, I attended a talk by Cat Allman at a meetup sponsored by Women Who Code. Cat is the manager of outreach for the Open Source Programs Office at Google and she spoke about how women can get into open source software. In particular, she emphasized OPW because it's one of the rare internship programs which is open to non-students. The only requirement is the applicant identifies as a woman, is at least 18, and has not previously participated in OPW or Google Summer of Code. This was especially good for me, since I am no longer a student. I spoke with Cat after her talk, and she strongly encouraged me to apply for OPW.

I didn't have enough time to do the winter 2013 round, but I made it my goal to apply for the summer 2013 round. After working hard on my application all of April 2013, here I am!

Top image is participation logo by L. Yu, Eller, Ford, Atanasoska, Muraus, and Zhurakhinskaya ↩
Outreach Program for Women main page ↩
OPW is for self-identified women only and doesn't require a project proposal. Also, one can only participate in OPW once. ↩
Most people only apply to one project at a single organization, but some apply to projects at multiple organizations to increase their odds of success. ↩

Demonstrating Pelican, a static site generator

2013-06-13T00:00:00-04:00

When I was setting up this blog, I initially started with Wordpress, but then I discovered that many (tech-savvy) people were using static site generators.

It took me some time to understand what that meant. Most of the common blogging platforms generate websites dynamically. Wordpress requires SQL to manage a database containing your blog content. When someone loads your blog URL, Wordpress generates the requested webpage. Wordpress must be installed on your webhost/server for this to work.

The idea behind static site generators is to have software on your local machine that creates webpages and then upload these pages to the webhost. The webpages have fixed content, hence the description "static site." You don't need to install anything on the webhost.

For a small personal blog, a static site makes a lot of sense because you

Avoid the security issues of having software (like Wordpress) installed on your webhost
Don't need to worry about upgrading and maintaining server-side software (like Wordpress)
Can write blog posts in some type of markdown, which is a lot more user friendly than composing posts in a browser GUI¹
Can exercise version control on the static web content
Can easily move your website somewhere else because your website is just a bunch of static files

In practice, I wanted my blog to have the following features:

Ability to write posts in markdown
Footnotes
Syntax highlighting, especially for code
Ability to render $\LaTeX$

I can show that all of that is possible in Pelican, a static site generator written in Python. (Needless to say, I'm using Pelican² for this blog.)

Markdown

Emphasis

This is extremely important.

Lists

Some things I'd like to do someday

Skate the canals in the Netherlands
Build my own computer
Write my own software project
Finish Baldur's Gate
Publish an essay

Top 5 console games

Paper Mario: Thousand Year Door (Gamecube)
Prince of Persia trilogy (The Sands of Time, Warrior Within, The Two Thrones) (console/PC)
Chrono Trigger (SNES/DS)
Phoenix Wright series (GBA/DS)
The Legend of Zelda: Twilight Princess (Gamecube/Wii)

Blockquote

Some quotes I like:

Literary

Having to read a footnote resembles having to go downstairs to answer the door while in the midst of making love.³

Self-reliance

A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.⁴

Images

Who can resist cute cats? (Yes, that is my photo and those are our cats.)

Horizontal rule

Syntax highlighting

Inline highlighting

Some Python keywords are lambda, import, and while.

Block of code

def factorial(n):
    if n == 0:
        return 1
    else:
        return n * factorial(n-1)

$\LaTeX$ symbols

Inline LaTeX symbols

I always seem to run out of Greek letters like $\alpha, \beta, \gamma$.

{equation} environment

$$ \begin{equation} i\hbar\frac{\partial}{\partial t}\Psi = \hat{H}\Psi \end{equation} $$

{eqnarray} environment

Do you know what these equations are?

$$ \begin{eqnarray} \nabla \cdot \vec{E} &=& 4\pi\rho \\\ \nabla \cdot \vec{B} &=& 0 \\\ \nabla \times \vec{E} &=& -\frac{1}{c}\frac{\partial \vec{B}}{\partial t} \\\ \nabla \times \vec{B} &=& \frac{1}{c}\left(4\pi\vec{J} + \frac{\partial \vec{E}}{\partial t}\right) \end{eqnarray} $$

The famous Maxwell's Equations⁵ describing electrodynamics in a vacuum!

For the eqnarray environment, I noticed that I had to use \\\ instead of \\ for the line breaks.

Source

The Markdown file used to generate this post is available here. I had to change the .md extension to .txt to avoid confusing the Pelican compiler.

Technical details

If you liked this demonstration of Pelican, you can see another post where I explain how I setup Pelican with full technical details.

I used to write my blog posts in HTML, in the blogging software's GUI. I constantly fought the software reformatting my edits and changing the whitespace. ↩
Top image taken from Pelican documentation page ↩
Attributed to Noel Coward ↩
From Time Enough to Love by Robert Heinlein ↩
In cgs units, of course! ↩

Hack for Western Mass

2013-06-05T00:00:00-04:00

This past weekend, I participated in Hack for Western Mass as part of the National Day of Civic Hacking.¹ This was my first hackathon and initially I was skeptical that you could accomplish much of anything with strangers during a weekend. But I turned out be wrong.

I was truly impressed with the ideas that community leaders and activists came up with including tracking the federal budget, developing a website to encourage local banking, mapping well sites in Western Mass, making a database/website to share information about local seeds. I worked on a project to map data on impoverished clients and reference this data with census data. A woman from the US Census came all the way from Connecticut to help out with the project, which illustrates how important this is. The census data was used to anonymize client information since we could map client addresses to specific census tracts, which was specific enough for data analysis, but reasonably obscured clients' personal data. This project was also my first experience working on a software team and it was interesting to listen to all the conversations. We had two experienced developers and since they understood project management, they really drove the conversation. In general, I noticed that across all the ideas, there was a common need for websites, databases, mapping, and visualization.

During the afternoon of the second day, the teams presented their work. People did great work for the short time period. The teams that made websites had the most polished projects, whereas teams that had a lot of backend work (like ours) seemed more likely to present incomplete work.

The number of interesting ideas and the enthusiasm of all involved was a great inspiration to me. Community building and grassroots activism is important to me and I was very glad to find out that hackers and developers can help these people out.

It appears that if you have front-end website/design skills or mapping/visualization skills, you can easily use these skills for social good! There are probably people in your community who need you!

Top image is official logo from the National Day of Civic Hacking Logo kit ↩

First post

2013-06-04T00:00:00-04:00

This is a new blog I’ve started for my work-related activities — which currently include software development, open source software, and data science.

My background is mainly in physics, though I also have an undergraduate degree in electrical engineering. I took the usual beginner programming courses and wrote scripts in Matlab and Python for scientific research.

I’m fairly new to software development, despite being a heavy user of software. This blog will help me keep track of software projects I’m working on and things I’ve learned.

I can't guarantee that I'll always be working on software and this blog may change in the future. But this is what it is, for now. Feel free to comment.

terriyu@/media/blog

New mobile friendly theme

Becoming a new contributor to open source through Outreach Program for Women and OpenStack

Initial thoughts

Applying to Outreach Program for Women

Working on OpenStack

The one important thing I learned about contributing to open source

How to install Devstack and Ceilometer on a Vagrant virtual machine

1. Install VirtualBox and Vagrant

Install VirtualBox

Install Vagrant

2. Configure Vagrant

Install Vagrant box

Create and configure Vagrantfile

3. Install Devstack and Ceilometer

Boot up the Vagrant VM

Install Devstack and Ceilometer

I already installed Devstack, but not Ceilometer

Troubleshooting

4. Check that Devstack is working

5. Configure Devstack for Ceilometer

6. Check that Ceilometer is collecting data

7. Run Ceilometer tests

More ways to run tests

Troubleshooting

8. References

Why OpenStack

How I setup Pelican

Demonstration

Installing Pelican and dependencies

Running pelican-quickstart

Fork and clone the GitHub repos for pelican-themes and pelican-plugins

Modify Pelican theme to taste

Make additional directories inside content directory

Modify settings in pelicanconf.py and publishconf.py

Setup Google Analytics/Piwik and Disqus

Boston WiSE Software Carpentry boot camp

Background

Preparations

Boot camp

Panel discussion

Problems, impressions, and anecdotes

Feedback

UMass Amherst Software Carpentry boot camp

First day of Outreach Program for Women

Demonstrating Pelican, a static site generator

Markdown

Emphasis

Lists

Blockquote

Images

Horizontal rule

Syntax highlighting

Inline highlighting

Block of code

\(\LaTeX\) symbols

Inline LaTeX symbols

{equation} environment

{eqnarray} environment

Source

Technical details

Hack for Western Mass

First post