I got to hang out and chat about debugging with the Ruby Rogues! I was totally flattered to be invited to be their guest for Ruby Rogues episode 150: The Debugging Mindset with Danielle Sucher, and had lots of fun recording the show.

It was so fantastic to just get to chat about science and problem-solving and trying to get better about putting our egos aside and really evaluating the evidence before us with such a great group of people.

It started like this…

DAVID: I bought a microscope yesterday. And there was a splotch on it and I couldn’t figure out what it is and I did the scientific method trying to figure out where in the microscope the splotch was coming from. Turns out, I was seeing a reflection of my optic nerve.

JAMES: Nice.

[Chuckles]

JOSH: Yeah, you can look in the microscope a really long time and you won’t find that.

DAVID: Yeah.

DANIELLE: So, when you gaze into the microscope, the microscope gazes back into you.

[Laughter]

DAVID: Also gazes back to me, yeah.

JOSH: [inaudible] Are you saying that what you see inside Dave’s eyes is the abyss?

DAVID: Yes.

DANIELLE: Yeah, yeah.

JAMES: I just want to know how he proved that hypothesis false. Did he gouge one of his eyes out?

[Laughter]

DAVID: Actually, and this is the part that I was very, very proud of, I finally switched eyes. And the splotch moved and changed shape.

So brilliant!

And this was my favorite quote of mine from the episode:

“Look, the goal is to prove that I’m wrong. That means I win. I’ve proved that I was stupid about something so I can move on to being stupid about something more interesting.”

Really, you can just check out the whole episode here. Have fun!

Me: “Remembering the names of things is the worst! Like, I can never remember which one is the trainwreck rule.”

Dave: “That’s the Law of Demeter.”

Me: “Right, I also can never remember which one the Law of Demeter is, so that makes sense. But I know and understand the actual principle!”

Dave: “Think of the dots as grains of wheat, and Demeter is the goddess of the harvest! Or think of the e’s in ‘Demeter’ as the dots in the trainwreck?”

Me: “Nah, but I can think of the e’s as regex dots and visualize the trainwreck as /D.m.t.r/! Though to be fair, that would also match Damatar, Dumutur, Dimitir…”

Dave: “Ooh, that works perfectly - with ancient Egyptian, when we don’t know what a vowel sound really was, ‘e’ is actually used as the default vowel!”

AND NOW IT’S PERFECTLY SITUATED IN MY MIND PALACE.

Next weekend is the 2014 MIT Mystery Hunt, and I’ve been going through Prolog puzzles to prep in eager anticipation. And since I have puzzles on the brain, and this past weekend was the last issue of 24 Magazine, it seems like the right moment to finally post the first cryptic crossword I ever wrote! (This is from back when I was working on 24mag issue 4 in February of last year.)

This latest and final issue of 24 Magazine is stunningly beautiful, rich with color and texture, and I am a little in love with it. I’m totally allowed to say that, because I didn’t work on this issue at all! But I’m incredibly proud of and impressed by my friends who did. You can read 24 Magazine issue 6 (the last issue ever!) online here.

So, cryptic crosswords! They’re a bit different from the usual sort of crossword you might find in the paper. Each clue actually has two parts - a meaning clue, and a wordplay clue. Common forms of wordplay used in Cryptic clues include (but are not limited to): anagrams, hidden words, double definitions, containers, and homophones. Oh, and you’ll never see the meaning clue in the middle of the wordplay clue, mind - it’ll always be at the beginning or the end.

Here’s a great example from the 2012 MIT Mystery Hunt:

“Charge or no charge, rotten root must be extracted (3)”

You can deconstruct it as follows: “Charge” is the definition, and “no charge, rotten root must be extracted” is the wordplay clue. “No charge” is FREE, from which R (“rotten root” - the first letter of ‘rotten’) is “extracted”. FREE minus R gets you to the answer: FEE.

The following is the first cryptic crossword I wrote, over the course of one long sleepless day and night of magazine construction. If the clues are too hard, I promise it’s my fault. Enjoy!

(Extra thanks to Dave Turner, Mike Develin, and Martin DeMello for test-solving, brainstorming, and generally playing along with me back in February 2013 when I was writing this.)

Tech-related books I loved reading in 2013

Emergent behavior

• Turtles, Termites, and Traffic Jams: Explorations in Massively Parallel Microworlds by Mitchel Resnick - Thoughts on experiments in emergent behavior using a Logo variant.
• Vehicles: Experiments in Synthetic Psychology by Valentino Braitenberg - I maybe kinda have a thing for the concept of emergent behavior. I blame having read Hofstadter at age 14-ish.

Design

• Practical Object-Oriented Design in Ruby by Sandi Metz - This was the sort of technical book you can both read on the train and actually get something valuable out of, which is a bit of a rare combination.
• Confident Ruby by Avdi Grimm - Guarding the borders.
• The Unix Philosophy by Mike Gancarz - “Every program written since the dawn of computing is a filter.”

Problem-solving

• Working Effectively with Legacy Code by Michael C. Feathers
• How to Solve It: A New Aspect of Mathematical Method by G. Polya - This is an extraordinary book. It claims to be about solving math problems, but of course it applies to problem-solving generally. No hugely new-to-me ideas, but I was mostly reading it for help articulating concepts when trying to teach debugging techniques, and it’s great for that.

History

• Engines of the Mind by Joel Shurkin - Includes lots of good stories about Ada Lovelace and von Neumann (why didn’t anyone ever tell me he was a Hungarian Jew with a penchant for dirty limericks?!).
• Coders at Work by Peter Seibel

Functional

• The Little Schemer by Daniel P. Friedman and Matthias Felleisen - Started off with a bunch of stuff I already knew (yeah yeah recursion whatevs), and then in the last maybe 30ish pages it suddenly sped up and got fascinating and brilliant and wonderful.
• Understanding Computation by Tom Stuart - Walks through automata, turing machines, lambda calculus, &c, writing interpreters and parsers along the way.
• Coffeescript Ristretto and Javascript Allonge by Reginald Braithwaite - Clear explanations of closures, combinators, &c.
• Learn You Some Erlang for Great Good! by Fred Hebert - Okay, I confess, I still haven’t actually finished reading this one yet, but I’d never written any Erlang before starting it and now I have, so it’s definitely been useful already.

Other

• The Tangled Web: A Guide to Securing Modern Web Applications by Michael Zalewski - Tremendously useful introduction to where things can go wrong.
• Learn Vimscript the Hard Way by Steve Losh
• The Unix Programming Environment by Brian Kernighan and Rob Pike - I’m only almost done with this at the moment, but I read most of it in 2013, so it totally still counts.
• A Unix Shell in Ruby by Jesse Storimer - Reading this made it finally click in my head what a shell is and isn’t, I think. The turtles are revealing themselves.
• Ruby Under a Microscope: An Illustrated Guide to Ruby Internals by Pat Shaughnessy - Wonderfully clear explanations of how MRI especially works. My favorite bits were the step-by-step explanations of C snippets along the way. Worth reading for the sake of learning about compilers, even if you’re not interested in Ruby.

Books I loved reading in 2013 that were emotionally difficult:

• Torture and Democracy by Darius Rejali - Riveting, intense, emotionally difficult. I would urge everyone to try to read this book, but I wouldn’t blame anyone for deciding it was too dark to handle. The history of torture and how different governments use different techniques, their goals, their lineage, and how public scrutiny has led to the proliferation of clean torture (that which does not leave marks) rather than lessening torture.
• In the Heart of the Sea: the Tragedy of the Whaleship Essex by Nathaniel Philbrick - I found that by the time the humans were dying of dehydration and starvation, I found it a bit hard to have sympathy for them, having just read so much detail about what they’d done while whale-hunting.
• The Glass Castle by Jeanette Walls - A novel that reminded me of some people I’ve known. This book tore me to pieces. Basically, it was a spectacular book that made me feel like I was going to throw up all the way through.
• The Testament of Jessie Lamb by Jane Rogers - Creepy as fuck novel. I’d stick it near The Handmaid’s Tale if my shelves were organized more organically. Themes of family and feminism and right to control over one’s own body.

Fiction I loved re-reading in 2013:

• A Deepness in the Sky by Vernor Vinge - The best of Vinge. Best read in quick succession with Elizabeth Moon’s The Speed of Dark.
• No one belongs here more than you. by Miranda July - Amazing short stories.
• Hereville: How Mirka Got Her Sword by Barry Deutsch - Still LOVE it. A comic book about an Orthodox Jewish girl fighting a troll.

Other Fiction I loved reading in 2013:

• Heiresses of Russ 2012: the year’s best lesbian speculative fiction edited by Connie Wilkins and Steve Berman - A bunch of truly fantastic stories!
• The James Tiptree Award Anthology 3 edited by Karen Jay Fowler, Pat Murphy, Debbie Notkin, and Jeffrey D. Smith - These are the best anthologies.
• The Quantum Thief by Hannu Rajaniemi - A bit Charles Stross, a bit Alistair Reynolds, a lot good.
• Emissaries From the Dead by Adam-Troy Castro - Decent scifi, made great by the presence of a character who is comprised of two linked humans who have transitioned into a single person and who has to deal with a lot of the same issues as trans people in our society.
• Civilwarland in Bad Decline by George Saunders - Short story and novella collection. His characters all have the same voice, but it’s a voice that really speaks to me. Marvelous satire, resignation, acknowledgment of futility, sharply hilarious and depressing all at once.
• Neptune’s Brood by Charles Stross - Financial mystery and space travel!
• Code Name Verity by Elizabeth Wein - I’m incredibly burnt out on Holocaust stories, and I still loved this one. (My grandparents were in Auschwitz. Ask me what my grandmother says about Dr. Mengele sometime, I dare you.)
• A Naked Singularity by Sergio de la Pava - A novel of philosophy and the NYC criminal justice system. Deeply nostalgic for me - this must’ve been written by someone who has actually spent time in the NYC criminal courts.

Other non-fiction I loved reading in 2013:

• Into Thin Air by Jon Krakauer - Memoir of disaster when climbing Everest.
• Positively Fifth Street by James McManus - Poker and murder. Good, though I think I would’ve enjoyed it more if I actually understood poker. It kinda makes me want to finally learn to play, though.
• Debt: The First 5,000 Years by David Graeber - I wouldn’t take this as a serious history, but I found it sociologically fascinating regardless.
• Central Park in the Dark: More Mysteries of Urban Wildlife by Marie Winn - I want to hang out in the park now and find owls and identify moths and watch slug sex!

Total number of books read in 2013: 123

I’ve been goofing around with Vim and Erlang lately, and since two great tastes taste great together, I made you a thing! My latest toy - Virex, where you can experiment with Vim’s regex on the web. It’s like Rubular but for Vim regex. Have fun!

I’ve been having a lot of fun mucking around with learning and tweaking my Vim-related toolset ever since I started working at Case Commons, so when a coworker asked why there wasn’t anything like Rubular for Vim’s regex, I jumped at the excuse to throw this together.

Using Virex is slower than just experimenting in Vim locally, natch, but it was fun to build and it comes in handy when you’re out with your friends arguing about regular expressions in Vim with only your phone handy for trying to prove your point. (SHUT UP, this happens.)

The interesting part turned out to be thinking about security, which is what the rest of this post will mostly be about.

Erlang is handy for sending messages between processes (shocking, n/n?)

I wanted to delegate the user-input test strings and regex patterns directly to Vim rather than try to reimplement Vim’s regex perfectly, while avoiding sending anything through the shell (danger zone like whoa, obvs). Erlang’s open_port/2 function was the perfect solution.

(Okay, I admit, I also really just wanted an excuse to play with Erlang some more. No Starch Press offered to send me free books to review a few months ago, and on a whim I asked for a copy of Learn You Some Erlang for Great Good!. I’ve been having a lot of fun exploring Erlang on the side ever since the book arrived.

It’s a pretty fantastic book, overall - concepts are explained clearly and thoroughly, in a way that I find very intuitive. My only caveat is that I found some of the examples used by the author distractingly offensive - I was really put off by bound variables being illustrated by a sadface dude in a suit standing next to a smiley lady in a white dress. Also, binary gender examples much? So, problematic. But “I like things, and some of those things are problematic.” It’s definitely also clear, thorough, and informative.)

Waaah don’t shell out via Vim please

So, great, user input is bypassing the shell when being sent directly from my Erlang server to Vim. But wait, it’s possible to shell out from Vim in various ways! Oh noes, we can’t have that.

I’m highlighting matches by using Vim’s regex substitution, %s/PATTERN/REPLACEMENT/g. The risky aspect of this is that the REPLACEMENT section can take any vimscript expression, so I don’t want users to be able to escape the PATTERN section and potentially get arbitrary code executed that could let them shell out and cause trouble.

This led to a truly absurd bit of Erlang that rejects any user-input pattern with a forward slash preceded by an odd number of contiguous backslashes. Tsk tsk, don’t go trying to escape my slash and causing trouble.

Regular Expressions Denial of Service attacks

The other big security risks I fretted over were Regular Expressions Denial of Service attacks. Regular expressions are pretty powerful, and can be written to run dangerously slowly and consume large amounts of memory.

I had a lot of fun testing Virex with this list of Evil Regexes I found on Wikipedia and this fabulous post Dave sent me, In search of an exponential time regex.

The biggest ReDoS problem Vim’s regex seems to be susceptible to is greedy quantifiers along the lines of \(.*\)\{1,32000\} - that hung forever. Bummer.

After a bit of poking around, I determined that \{99,\} and \{,99\} were safe, but \{999,\} and \{,999\} are not. So, Virex rejects repetitions that are 3 digits are longer.

Here’s the function I’m using to test whether a user-input regex pattern is safe:

safe(Pattern) when erlang:length(Pattern) > 80 ->
  false;
safe(Pattern) ->
  DangerousRegex = "\{-?[0-9]{3,}|[0-9]{3,}\\\\?\}|([^\\\\]|^)(\\\\\\\\)*/",
  re:run(Pattern, DangerousRegex) =:= nomatch.

As you can see, I’m limiting patterns to 80 characters on basic principle - if you need to test a longer regex than that, you can do it when you get home.

If the pattern is short enough, that long regex I’ve got there does two other checks - it makes sure that no quantifiers have repeats that are 3 digits are longer, and that no forward slashes are immediately preceded by an even number of backslashes.

(Why so many backslashes? Blame Erlang. I feel like half the time I spent on this little project was focused on making sure I was escaping characters properly as they went through Erlang, Vim, and oh god you got your regex in my regex.)

So.

To sum up - Virex is a webmachine app, with nginx acting as a reverse proxy and serving the static content, which sanitizes the user-input regex patterns and sends them off to Vim to test them out. Alex Feinman designed that awesome logo for me. The source code is here.

I adore Erlang’s syntax, and I had a lot of fun exercising the paranoid portion of my brain and exploring evil regexes - hopefully I caught them all. If you can think of anything else I ought to test for, please let me know! (Ideally via twitter or pull requests, not by crashing my server, thankyouverymuch. ^^)