I've just finished my TechEd 2011 presentation on advanced .NET debugging. I covered using WinDBG and SOS to troubleshoot memory leaks, deadlocks, race conditions.
You can download the powerpoint presentation here
Wednesday, August 24, 2011
Sunday, July 31, 2011
Helper class for creating memory dumps of a Managed Process
I'm giving a talk at TechEd NZ 2011 in about a month. As part of that talk, I'll mention creating memory dumps using the MiniDumpWriteDump function, and show a helper class which P/Invokes it
Here is that helper class (Updated 19 Sept 2011 to fix some bugs).
Here is that helper class (Updated 19 Sept 2011 to fix some bugs).
using System.Runtime.InteropServices;
using System;
using System.IO;
using System.Diagnostics;
using System.Threading;
public static class DbgHelp
{
[StructLayout(LayoutKind.Sequential, Pack = 4)]
struct MINIDUMP_EXCEPTION_INFORMATION
{
public uint ThreadId;
public IntPtr ExceptionPointers;
[MarshalAs(UnmanagedType.Bool)]
public bool ClientPointers;
}
[DllImport("Dbghelp.dll")]
static extern bool MiniDumpWriteDump(
IntPtr hProcess,
uint ProcessId,
IntPtr hFile,
[MarshalAs(UnmanagedType.I4)] MiniDumpType DumpType,
IntPtr ExceptionParam, // Ptr to MINIDUMP_EXCEPTION_INFORMATION
IntPtr UserStreamParam,
IntPtr CallbackParam);
[DllImport("kernel32.dll")]
static extern uint GetCurrentThreadId();
enum MiniDumpType : int
{
MiniDumpNormal = 0x00000000,
MiniDumpWithDataSegs = 0x00000001,
MiniDumpWithFullMemory = 0x00000002, // required for .NET apps
MiniDumpWithHandleData = 0x00000004,
MiniDumpFilterMemory = 0x00000008,
MiniDumpScanMemory = 0x00000010,
MiniDumpWithUnloadedModules = 0x00000020,
MiniDumpWithIndirectlyReferencedMemory = 0x00000040,
MiniDumpFilterModulePaths = 0x00000080,
MiniDumpWithProcessThreadData = 0x00000100,
MiniDumpWithPrivateReadWriteMemory = 0x00000200,
MiniDumpWithoutOptionalData = 0x00000400,
MiniDumpWithFullMemoryInfo = 0x00000800,
MiniDumpWithThreadInfo = 0x00001000,
MiniDumpWithCodeSegs = 0x00002000,
MiniDumpWithoutAuxiliaryState = 0x00004000,
MiniDumpWithFullAuxiliaryState = 0x00008000,
MiniDumpWithPrivateWriteCopyMemory = 0x00010000,
MiniDumpIgnoreInaccessibleMemory = 0x00020000,
MiniDumpWithTokenInformation = 0x00040000
};
public static void WriteExceptionDump(string filePath)
{
var proc = Process.GetCurrentProcess();
int win32Error;
if(!TryCreateDump(filePath, proc.Handle, (uint)proc.Id, GetCurrentThreadId(), Marshal.GetExceptionPointers(), out win32Error))
throw new Exception("Couldn't create dump file! Error: 0x" + win32Error.ToString("X8"));
}
public static bool TryCreateDump(string dumpFilePath, IntPtr processHandle, uint processId, uint threadId, IntPtr exceptionPointers, out int win32Error)
{
bool success = false;
int lastError = 0;
// Dump on a seperate thread - IsBackground=false is important to stop the process exiting while we write the dump
// also works around an issue of VS not being able to walk the callstack of the crashing thread
var thread = new Thread(new ThreadStart(() => {
// In-process dumps must ClientPointers = false
// If ClientPointers is false, or if there are no ExceptionPointers we must pass IntPtr.Zero as ExceptionInfo
var exceptionParam = IntPtr.Zero;
if (processId != Process.GetCurrentProcess().Id && exceptionPointers != IntPtr.Zero)
{
var ei = new MINIDUMP_EXCEPTION_INFORMATION {
ClientPointers = true, // in-process dump. True if we're dumping external processes
ExceptionPointers = exceptionPointers, // may be IntPtr.zero for CLR exceptions
ThreadId = threadId,
};
exceptionParam = Marshal.AllocHGlobal(Marshal.SizeOf(ei));
Marshal.PtrToStructure(exceptionParam, ei);
}
using (var outputFile = new FileStream(dumpFilePath, FileMode.Create))
{
success = MiniDumpWriteDump(
processHandle,
processId,
outputFile.SafeFileHandle.DangerousGetHandle(),
MiniDumpType.MiniDumpWithFullMemory,
exceptionParam,
IntPtr.Zero,
IntPtr.Zero);
}
if (!success)
lastError = Marshal.GetLastWin32Error();
if (exceptionParam != IntPtr.Zero)
Marshal.FreeHGlobal(exceptionParam);
})) { IsBackground = false, Name = "MiniDump thread" };
thread.Start();
thread.Join();
win32Error = lastError;
return success;
}
}
Sunday, January 10, 2010
DNUG Reactive Framework Presentation
The talk seemed to go down well - a PDF export of the slides from KeyNote is available if you're interested
Here's a brief outline of what I talked about:
Note: I had a recurring theme throughout the presentation that shorter code is better. I had random slides thrown in with quotes from programming "celebrities" to keep trying to push the point. I did this because I personally believe in it, and also because being able to do more with less code is one of the big benefits of the Reactive Framework.
As I've met more than a few developers who are still running .NET 2.0 on VS2005, I started by giving a brief recap on the C# 3.0 language features that Rx makes heavy use of (lambdas, linq, and so forth)
I then did a quick overview defining exactly what Asynchronous programming is, and why you'd want to do it (Rx is all about asynchronous programming after all)
With the overviews out of the way, I talked a bit about the IObservable interface and how it related to IEnumerable, and showed a few short code snippets of code using Rx (looking surprisingly the same as ordinary Linq), and some diagrams showing the timeline of things that happen when using both IEnumerable and IObservable
I then cut to a demo. Sample code shown in the demo is available as on Google Code and I'm placing it under the creative commons by attribution license. You don't have to provide attribution (it's a code sample!), but I can't find a CC license other than public domain that doesn't require attribution.
Most of the demo focused on the WcfClient and WcfServer projects - they're the most interesting, so I'd suggest looking at those if you're interested.
I followed up with a few more slides containing other tidbits (such as the fact that you get a backport of the .NET 4 parallel task library for free with Rx), and some links.
Enjoy!
Saturday, March 28, 2009
Programming podcast roundup
When the stackoverflow podcast first launched, I downloaded it and gave it a listen. I enjoyed it, and I was sick of listening to the same old music when going running... and so, my podcast-listening-habit was born.
Thus far I've been stuck in the microsoft-centric technology podcasts. This not because I'm a microsoft shill, but because I haven't been able to find any non-microsoft-centric podcasts out there.
At any rate, here's the ones I regularly listen to, ranked by preference.
1. The Stack Overflow Podcast
Admittedly I'm biased as this was the first podcast I listened to, and I've been following it since day one. Even had this not been the case, I think I'd still rank it highly.
The SO podcast primarily consists of Jeff Atwood and Joel Spolsky chatting about the stackoverflow site, and programming/IT topics in general. Major topics are either pulled from the SO site, from reader questions, or often based on current events, or whatever they're each up to.
Even if you think Joel and Jeff are a pack of jumped up blowhards (as many no doubt do), they're still really entertaining to listen to. Both speak well and have a wide variety of experience to draw on (Joel in particular is the king of 'back in my day' type stories, which are usually very interesting). They also complement eachother well which makes for good listening.
I like the fact that the show is centered around them and on the stackoverflow site. They'll occasionally have third party participants on the show, but the majority of shows are just Jeff and Joel. I find this helps you feel like you "know them" better, rather than that they're just interviewers or reporters.
Finally, the SO podcast gets a big bonus for not being full of annoying ads. It has a small bit at the beginning and end from IT conversations, who provide their hosting, and that's it. It typically runs for about an hour.
2. Hanselminutes
Hanselminutes is Scott Hanselman interviewing people about technology. Every week there's a different guest, always talking about some recent technology. Most (but not all) are microsoft based, which is no doubt a side effect of Scott being a microsoft employee.
This does however have it's upsides, as you'll get to find out stuff by hearing it directly from other microsofties that come on the show, rather than hearing things through the rumourmill / blogosphere / reddit.
Hanselminutes is very well produced, and Scott really knows how to do a good interview. Most of the guests are top-notch, and often the tech talk gets pretty deep, which IMHO is great, as it provides the substance of the show.
Recently the podcasts took a bit of a detour, while Scott was in south africa - and interviewed some of the local people about non-technical things, and his family (his wife is from Zimbabwe). I really enjoyed these, as it was really cool to get a bit of insight into the way things are in some other (non-westernised) countries.
Hanselminutes has some advertising at the beginning, and typically has a single "spliced in" ad in the middle. These tend not to be too long though. The show usually runs for half an hour.
3. The Australian Gamer Podcast
OK, this is not programming related at all, but I'm an ex gamer so I like to keep up with that scene periodically.
At any rate, Matt and Yug (the hosts) are _very_ funny (in a crude guy-humour kind of way. It's R18, you have been warned.)
I know of many people who have little to no interest in cars, yet enjoy watching Top Gear, because the presenters simply put on a really great entertaining show.
In my humble opinion at least, the AG podcast is similar. I played it in the car while driving somewhere with my girlfriend (who is not a gamer in the slightest), and she said that apart from all the swearing, it was actually pretty good. That's high praise :-)
4. Herding Code
Herding code is a "technology round-table" run by K. Scott Allen, Kevin Dente, Scott Koon and Jon Galloway, who are all either microsofties, MVP's, or otherwise working using the microsoft technology stack. It's usually just them sitting around chatting (4 people is more than enough to keep a conversation going), with occasional interviews.
It's not as professionally produced as some other podcasts, but it does have a really good friendly atmosphere. You feel like these guys are good mates, and they'd be sitting around talking about this stuff anyway, recording or not.
I really like this, as you feel like you get to know them a lot more, which keeps you engaged.
Herding code doesn't have any sponsored introductions, or inline ads at all. This is awesome, but I wonder if it's just because the show hasn't picked up any yet?
5. Deep Fried Bytes
DFB is another interview driven show, run by Chris Woodruff and Keith Elder. It's kind of similar to hanselminutes, in that there's a guest being interviewed or talked to every show, but Chris and Keith bring their own brand of humour and atmosphere, and also have great character. The "feel like you know them" factor is high.
My one complaint about DFB is that the format thus far seems to be like this: 1) Go to a conference and record interviews with conference speakers and attendees. 2) publish interview as a podcast, repeat until you've run out of interviews, then go to another conference
This is not in and of itself bad, but when you're putting out shows in february which were recorded at the PDC in october, it starts to wear thin a bit.
DFB doesn't have any sponsored intros or ads, and runs for around 45 minutes.
6. .NET Rocks!
DNR is the granddaddy of the tech podcast. Carl Franklin and Richard Campbell are currently on show Number 432, and put out new ones every week like clockwork.
It's another interview-centric show, focused around the Microsoft .NET ecosystem. It sometimes has a bit of an overlap with Hanselminutes (the same guests will appear on each show in quick succession when a new technology is coming out of MS), but it's different enough to always be worth listening to both.
The production quality is top notch (the best out of all the podcasts I've heard). Carl is a musician, and his obvious knowledge of all things audio shows itself here. Apart from the fact that they're talking about .NET programming, you could easily believe this was a show coming from your local radio station's best morning dj crew.
I feel kind of bad putting DNR down the bottom of the list, as it really is a very good podcast, but sadly it is the one I will listen to after I've heard all the others. This is more a testament to how much I like the others than an indictment against DNR. Carl and Richard have been doing this for a long long time, and they're very good at it.
The main thing that bugs me about DNR is actually the ads. They have a 2 minute sponsored intro, then usually 2 or 3 ads spliced into the middle of the show. The ads tend to be longer than on other podcasts, and more intrusive. Shows tend to run between 45 minutes to an hour, but if you skip the ads and the leadout, it's about ten minutes less. If you know of any other good tech podcasts, just drop a comment! Cheers, Orion
Monday, March 09, 2009
Duplicate Line in Visual Studio
Visual Studio ships with many features built in. "Duplicate the current line" doesn't appear to be one of them for some strange reason.
CodeRush Express ships with a "duplicate line" function, but it's WAY too clever for it's own good. It tries to work out whether you're duplicating a line with a variable, function, etc, and act accordingly. If it can't understand your line (perhaps it's just a string), then it FAILS. Unfortunately it only understands about 40% of actual lines of code, so this severely limits it's usefulness.
This is stupid. I just want the equivalent of "copy/paste the current line be done with it", so without further ado, here's a macro to do it. You can then bind a keyboard shortcut to the macro, and get on with more important things.
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics
Public Module Misc
Sub Duplicate_Line()
DTE.UndoContext.Open("Duplicate Line(s)")
Try
Dim ts As TextSelection = DTE.ActiveDocument.Selection
Dim epStart As EditPoint = ts.TopPoint.CreateEditPoint()
Dim epEnd As EditPoint = ts.BottomPoint.CreateEditPoint()
Dim lineText As String = epStart.GetLines(epStart.Line, epEnd.Line + 1)
epEnd.EndOfLine()
epEnd.Insert(Environment.NewLine)
epEnd.Insert(lineText)
ts.MoveToLineAndOffset(epEnd.Line, epStart.LineCharOffset())
Finally
DTE.UndoContext.Close()
End Try
End Sub
End Module
PS: Why can't I write VS macros in C#? VB just looks so ugly :-(
Monday, December 29, 2008
Windows 7 beta 1: sound does not work on Macbook Pro (RealTek HD Audio)
This post is googlebait: I couldn't find the solution for this on google, so here's how I solved it. Hopefully others will be spared the messing around.
After installing windows 7 beta 1 (7000) on my macbook pro, and installing the bootcamp drivers off the Leopard Disc, as well as The vista 2.1 bootcamp update, everything worked very nicely... Except sound.
Win7 detected "High Definition Audio Device" and everything looked like it should have worked, but no sound came out of the speakers.
After much mucking around, here's what I did:
Go into the leopard drivers folder. There should be a directory called Drivers, and under that is a file called RealTekSetup.exe. If you try run this normally, it will fail.
What I did next was:
As for win7 itself? Well, the beta is faster, nicer, and all around better than vista. I'll never go back. They didn't do a good enough job of copying the dock... but it's still miles ahead of vista, and that's another blog post.
Byebye!
Monday, September 29, 2008
Embedded IronRuby interactive console
Screenshot!
What this is, is a small dll which you can add to any .net winforms project. When run, it brings up the interactive console, and you can poke around with your app. It's running live inside your process, so anything your app can do, it can do. I thought this was kind of cool :-)
How to get it going:
- Download and build IronRuby by following the instructions on IronRuby.net - I built this against IronRuby SVN revision 153. As of RIGHT NOW the current revision is 154 which doesn't build.
- Download the Embedded IronRuby project from the following URL - you can use SVN to check it out directly from there. (I'm assuming familiarity with SVN in the interests of brevity)
http://code.google.com/p/orion-edwards-examples/source/browse/#svn/trunk/dnug/ironruby-presentation/EmbedIronRuby - Open the EmbeddedIronRuby/EmbeddedIronRuby.sln file in visual studio, and remove/add reference so that it references
IronRuby.dll,Microsoft.Scripting.dll,Microsoft.Scripting.Core.dll, andIronRuby.Libraries.dll. These will be in the IronRubybuild\debugfolder that you will have built in step 1. - Compile!
- For some reason, when you compile, Visual Studio will only copy
IronRuby.dll,Microsoft.Scripting.dllandMicrosoft.Scripting.Core.dllto thebin\debugdirectory. It also needsIronRuby.Libraries.dllin that directory (or in the GAC) to run, otherwise you get a stack overflow in the internal IronRuby code when you run it.
The joys of alpha software I guess :-) - Run the app and click the button!
EmbeddedRubyConsole.dll, and in your app do this:
new EmbeddedRubyConsole.RubyConsoleForm().Show();
Credit: Some of the 'plumbing' code (the TextBoxWriter and TextWriterStream) come from the excellent IronEditor application. Full credit to and copyright on those files to Ben Hall. Thanks!
IronRuby Presentation!
I recently gave a presentation to my local .NET user group about IronRuby.
Click on the image to download the slides as a PDF file.
Note: This was exported from keynote with speaker notes, which I've revised slightly since giving the presentation.
As part of this, I demoed a small library I wrote which gives you a live interactive ruby console as part of your running app.
Basically it lets you poke around your program and modify things while it's running. I'll post the code and notes about that shortly
Tuesday, July 29, 2008
Ruby Unit Converting Hash
I'm currently working on a project where I need to convert from things in one set of units to any other set of units ( eg centimeters to inches and so forth)
I had a bunch of small helper functions to convert from X to Y, but these kept growing every time we needed to handle something which hadn't been anticipated.
This kind of thing is also exponential, as if we have 4 'unit types' and we add a 5th one, we need to add 8 new methods to convert each other type to and from the new type
A few hours of refactoring later, I have this, which I think is kind of cool, and will enable me to delete dozens of small annoying meters_to_pts methods all over the place.
Disclaimer: This is definitely not good OO. A hash is not and never should be a unit converter. In the production code I will refactor this to build an actual Unit Converter class which stores a hash internally :-)
# Builds a unit converter object given the specified relationships
#
# converter = UnitConverter.create({
# # to convert FROM a TO B, multiply by C
# :pts => {:inches => 72},
# :inches => {:feet => 12},
# :cm => {:inches => 2.54,
# :meters => 100},
# :mm => {:cm => 10},
# })
#
# You can then do
#
# converter.convert(2, :feet, :inches)
# => 24
#
# The interesting part is, it will follow any links which can be inferred
# and also generate inverse relationships, so you can also (with the exact same hash) do
#
# converter.convert(2, :meters, :pts) # relationship inferred from meters => cm => inches => pts
# => 5669.29133858268
#
class UnitConverter < Hash
# Create a conversion hash, and populate with derivative and inverse conversions
def self.create( hsh )
returning new(hsh) do |h|
# build and merge the matching inverse conversions
h.recursive_merge! h.build_inverse_conversions
# build and merge implied conversions until we've merged them all
while (convs = h.build_implied_conversions) && convs.any?
h.recursive_merge!( convs )
end
end
end
# just create a simple conversion hash, don't build any implied or inverse conversions
def initialize( hsh )
merge!( hsh )
end
# Helper method which does self.inject but flattens the nested hashes so it yields with |memo, from, to, rate|
def inject_tuples(&block)
h = Hash.new{ |h, key| h[key] = {} }
self.inject(h) do |m, (from, x)|
x.each do |to, rate|
yield m, from, to, rate
end
m
end
end
# Builds any implied conversions and returns them in a new hash
# If no *new* conversions can be implied, will return an empty hash
# For example
# {:mm => {:cm => 10}, :cm => {:meters => 100}} implies {:mm => {:meters => 1000 }}
# so that will be returned
def build_implied_conversions
inject_tuples do |m, from, to, rate|
if link = self[to]
link.each do |link_to, link_rate|
# add the implied conversion to the 'to be added' list, unless it's already contained in +self+,
# or it's converting the same thing (inches to inches) which makes no sense
if (not self[from].include?(link_to)) and (from != link_to)
m[from][link_to] = rate * link_rate
end
end
end
m
end
end
# build inverse conversions
def build_inverse_conversions
inject_tuples do |m, from, to, rate|
m[to][from] = 1.0/rate
m
end
end
# do the actual conversion
def convert( value, from, to )
value * self[to][from]
end
end
I'm not sure if deriving it from Hash is the right way to go, but it basically is just a big hash full of all the inferred conversions, so I'll leave it at that.
Update
Woops, this code requires 'returning' which is part of rails' ActiveSupport, and an extension to the Hash class called recursive_merge!, which I found on an internet blog comment somewhere (so it's only fitting that I share back with this unitconverter)
Code for recursive_merge
class Hash
def recursive_merge(hsh)
self.merge(hsh) do |key, oldval, newval|
oldval.is_a?(Hash) ?
oldval.recursive_merge(newval) :
newval
end
end
def recursive_merge!(hsh)
self.merge!(hsh) do |key, oldval, newval|
oldval.is_a?(Hash) ?
oldval.recursive_merge!(newval) :
newval
end
end
end
Code for returning
class Object
def returning( x )
yield x
x
end
end
Monday, July 14, 2008
HaveBetterXpath
I'm rspeccing some REST controllers which return XML, and wanting to use XPath to validate the responses.
I came across this
http://blog.wolfman.com/articles/2008/01/02/xpath-matchers-for-rspec
Thanks to him. It worked nicely (couldn't be bothered messing about with hpricot to get that to go), but I didn't like the API as much as I could have.
Example of that API:
response.body.should have_xpath('/root/node1')
response.body.should match_xpath('/root/node1', "expected_value" )
response.body.should have_nodes('/root/node1/child', 3 )
I didn't like the fact that there were 3 distinct matchers, and that match_xpath didn't work with regexes. I re-worked it, so the API is now
response.body.should have_xpath('/root/node1')
response.body.should have_xpath('/root/node1').with("expected_value") # can also pass a regex
response.body.should have(3).elements('/root/node1/child') # Note actually extends string class and uses normal rspec have matcher
Extending the String class to support elements(xpath) is a win also because it lets you do things like
response.body.elements('/child').each { |e| more complex assert for e here }
Without further ado, new code here:
# Code borrowed from
# http://blog.wolfman.com/articles/2008/01/02/xpath-matchers-for-rspec
# Modified to use one matcher and tweak syntax
require 'rexml/document'
require 'rexml/element'
module Spec
module Matchers
# check if the xpath exists one or more times
class HaveXpath
def initialize(xpath)
@xpath = xpath
end
def matches?(response)
@response = response
doc = response.is_a?(REXML::Document) ? response : REXML::Document.new(@response)
if @expected_value.nil?
not REXML::XPath.match(doc, @xpath).empty?
else # check each possible match for the right value
REXML::XPath.each(doc, @xpath) do |e|
@actual_value = e.is_a?(REXML::Element) ?
e.text :
e.to_s # handle REXML::Attribute and anything else
if @expected_value.kind_of?(Regexp) && @actual_value =~ @expected_value
return true
elsif @actual_value == @expected_value.to_s
return true
end
end
false # our loop didn't hit anything, mustn't be there
end
end
def with_value( val )
@expected_value = val
self
end
alias :with :with_value
def failure_message
if @expected_value.nil?
"Did not find expected xpath #{@xpath}"
else
"The xpath #{@xpath} did not have the value '#{@expected_value}'\nIt was '#{@actual_value}'"
end
end
def negative_failure_message
if @expected_value.nil?
"Found unexpected xpath #{@xpath}"
else
"Found unexpected xpath #{@xpath} matching value #{@expected_value}"
end
end
def description
"match the xpath expression #{@xpath}, optionally matching it's value"
end
end
def have_xpath(xpath)
HaveXpath.new(xpath)
end
# Utility function, so we can do this:
# response.body.should have(3).elements('/images/')
class ::String
def elements(xpath)
REXML::XPath.match( REXML::Document.new(self), xpath)
end
alias :element :elements
end
end
end
Monday, July 07, 2008
How to: load the session from a query string instead of a cookie
We use SWFUpload to upload some images in a login-restricted part of the site.
There is a problem however, in that we weren't able to get SWFUpload to send the normal browser cookie along with it's HTTP file uploads, so the server couldn't tell which user was logged in.
The 'normal' solution to this is to add the session key to the query string, and have the server load the session from the query string if the cookie isn't present, only ruby/rails doesn't support doing that.
a nice guy with the handle 'mcr' in #rubyonrails on irc.freenode.org worked out how to make this work, by patching ruby's cgi/session.rb
Instructions
- Copy cgi/session.rb out of your ruby standard library into your rails app's lib folder
- explicitly load the file out of lib, which will then overwrite the built in code
Needless to say this will stop working if the ruby standard library version of cgi/session changes, but I don't see that as being very likely
Patch in unified diff format:
--- /usr/lib/ruby/1.8/cgi/session.rb 2006-07-30 10:06:50.000000000 -0400
+++ lib/cgi/session.rb 2008-07-07 21:07:12.000000000 -0400
@@ -25,6 +25,9 @@
require 'cgi'
require 'tmpdir'
+require 'tempfile'
+require 'stringio'
+require 'strscan'
class CGI
@@ -243,6 +246,20 @@
# undef_method :fieldset
# end
#
+ def query_string_as_params(query_string)
+ return {} if query_string.blank?
+
+ pairs = query_string.split('&').collect do |chunk|
+ next if chunk.empty?
+ key, value = chunk.split('=', 2)
+ next if key.empty?
+ value = value.nil? ? nil : CGI.unescape(value)
+ [ CGI.unescape(key), value ]
+ end.compact
+
+ ActionController::UrlEncodedPairParser.new(pairs).result
+ end
+
def initialize(request, option={})
@new_session = false
session_key = option['session_key'] || '_session_id'
@@ -253,6 +270,7 @@
end
end
unless session_id
+ #debugger XXX
if request.key?(session_key)
session_id = request[session_key]
session_id = session_id.read if session_id.respond_to?(:read)
@@ -260,6 +278,12 @@
unless session_id
session_id, = request.cookies[session_key]
end
+
+ unless session_id
+ params = query_string_as_params(request.query_string)
+ session_id = params[session_key]
+ end
+
unless session_id
unless option.fetch('new_session', true)
raise ArgumentError, "session_key `%s' should be supplied"%session_key
Sunday, July 06, 2008
How to: Avoid getting your database wiped when migrating to rails 2.1
We recently migrated some projects from rails 1.2 to 2.1.
In doing this, we encountered a bug where sometimes (in production only) running rake db:migrate goes wrong, and re-runs all your migrations
The unhappy side effect of it re-running ALL the migrations, is that it effectively re-creates your entire database, and you lose all your data. USEFUL
I didn't have the time or the luxury to figure out quite why this was happening, if anyone does, please comment and let me know what it was. Apparently there's been a few other blogs mentioning it, but I don't have any of them at hand.
The workaround is to manually create the schema_migrations table before you run rake db:migrate in rails 2.1.
If you put the following script in your RAILS_ROOT/db directory, and run it, it will do that.
Enjoy. (Disclaimer: if there's a bug in the script, and it does anything awful, it's not my fault! You have been warned!)
require File.dirname(__FILE__) + '/../config/environment'
# Define some models
class SchemaInfo < ActiveRecord::Base
set_table_name 'schema_info'
end
class SchemaMigration < ActiveRecord::Base; end
# Create the schema_migrations table
ActiveRecord::Migration.class_eval do
create_table 'schema_migrations', :id => false do |t|
t.column :version, :string, :null => false
end
end
# Work out the migrated version and populate the migrations table
v = SchemaInfo.find(:first).version.to_i
puts "Current schema version is #{v}"
raise "Version number doesn't seem right!" if v == 0
1.upto(v) do |i|
SchemaMigration.create!( :version => i )
puts "Added entry for migration #{i}"
end
# Drop the schema info table, as rails-2.1 won't automatically do it thanks to our hacking
ActiveRecord::Migration.class_eval do
drop_table 'schema_info'
end
How To: Create old rails apps when you have newer gems installed
My dev server has the gems for rails 1.2.6, 2.0.2 and 2.1.0 all installed.
You can see which ones you have by runninggem list --local | grep rails
The problem is, when I create new rails apps, it always uses the latest version. If I explicitly want to create a 1.2.6 or 2.0.2 app, then I can do it like this
rails _1.2.6_ some_old_app
Useful.
For the technically nosey, we can see how this works by reading the source of /usr/bin/rails, which is here
require 'rubygems'
version = "> 0"
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
version = $1
ARGV.shift
end
gem 'rails', version
load 'rails'
How to: Rails 2.0 and 2.1 resources with semicolons
Rails 1.X used semicolons as method seperators for resources, so you'd get
http://somesite/things/1;edit
Rails 2.X switches this to
http://somesite/things/1/edit
This is nice and all, but some of us have actual client applications which we can't all just upgrade instantly
To make the semicolon-routes still work in rails 2.X, so you don't break all your clients, do this
At the TOP of routes.rb, before the ActionController::Routing::Routes.draw block
# Backwards compatibility with old ; delimited routes
ActionController::Routing::SEPARATORS.concat %w( ; , )
and, at the BOTTOM of routes.rb BEFORE the end
# Backwards compatibility with old ; delimited routes
map.connect ":controller;:action"
map.connect ":controller/:id;:action"
Profit!
Monday, June 30, 2008
Failfox 3
Firefox 3 is great
BUT. I like to bookmark things by dragging from the URL bar to (a folder in) the bookmarks toolbar.
Look what happens in FF3.
You can't drag a bookmark onto a tooltip, so the whole thing fails.
(@*#^$)(*&@#)($*&@#(*$&@(#*$#@
Thursday, January 10, 2008
Paragon NTFS for mac update
In the comments of my last blog about the quick hack benchmark I did of paragon NTFS for mac OS X, Anatoly, the product manager from paragon replied. I'm reposting it here so it's not hidden behind that tiny little '1 comments' link at the bottom of the post.
Dear Orion, My name is Anatoly.
I am Product Manager for Paragon NTFS for Mac OS X driver.
Thank you for your time and efforts to measure the performance of Paragon NTFS for Mac OS X driver.
Frankly speaking your results are not exactly correct for real time usage of the driver.
First of all, the Finder application handles files (copy, create,...) using 2MB block size rather than 512B you tested (the "dd if=//tmp/bigfile of=/dev/null" command uses 512KB block size by default).
Second, to get precise figures you have to unmount/mount partitions every time you perform any test (the reason you got - 87.45MB/Sec).
So, we retested our driver and would like to show you our results.
We used commands that are similar to yours:
For write: dd if=/dev/random of=/Volumes/bigfile bs=2m count=100
For read: dd if=/Volumes/bigfile of=/dev/null bs=2m
HFS+ Firewire: Write (MiB/sec) - 4,26; Read (MiB/sec) - 36,06.
NTFS Firewire: Write (MiB/sec) - 4,24; Read (MiB/sec) - 35,26.
Please note in case we will use "bs=1m" we get:
HFS+ Firewire: Write (MiB/sec) - 4,34; Read (MiB/sec) - 39,29.
NTFS Firewire: Write (MiB/sec) - 4,30; Read (MiB/sec) - 42,25.
According to our tests we can assert that our driver has the same performance as the native HFS+ driver has.
Let me know if I am wrong.
Thank you, Anatoly.
Well, Wow. I always feel special when important people from companies reply to me! If anyone is looking for numbers, use those ones, as he obviously is far more clued up about it than I am.
I completely agree with his assertion that the driver performs as well as native HFS+
At any rate, I'd already purchased the product, and it's been great. If you are like me and need to access NTFS drives from your mac, you really should buy it.
Thanks!
I completely agree with his assertion that the driver performs as well as native HFS+
At any rate, I'd already purchased the product, and it's been great. If you are like me and need to access NTFS drives from your mac, you really should buy it.
Thanks!
Sunday, November 25, 2007
5 minute performance picture: Paragon NTFS for Mac OS X
I have a macbook pro, and a large amount of files, and I like to play computer games.
So, I have a large external firewire/USB2 hard drive, and boot camp.
This also means I care about NTFS access from OSX.
I'd been running MacFuse + NTFS3G. The performance was not toooo bad, but it was chock full of bugs. Drives would show up as network drives, and be called "-n External" and "-n" instead of "External". Not to mention that sometimes stuff would just randomly break. Files would sometimes disappear or move around in the finder and sometimes I just simply couldn't mount the drive. It sucked pretty hard, so I ended up booting up vmware and accessing that drive via vmware's USB2 mapping + samba under the windows VM. Not cool
Suffice to say I was very happy when I saw the release of Paragon NTFS for OSX
I downloaded it, got rid of MacFUSE and NTFS3G, and ran some benchmarks.
Before the benchmark results, let me first say that even if it was just as slow as MacFUSE/NTFS3g, Paragon NTFS would still be worth a look, because it seems (so far) to be rock solid. Drives show up as proper drives in the finder. The volume labels are fine, as is everything else I can see. There is no lag, and I even now have the option of backing up my boot camp partition with Time Machine. Basically it's as if Apple had actually bothered to implement full NTFS support in leopard. That's cool.
Anyway, Benchmarks:
To get the write speeds, I did this:
dd if=/dev/random of=//tmp/bigfile bs=1m count=200
For the read speeds, I did this:
dd if=//tmp/bigfile of=/dev/null
Yes I am aware this is a crap method of benchmarking drives/filesystems. I'm not anandtech and I don't have days to do this.
Computer: MacBook Pro 2.2ghz (the cheapest one)
External NTFS drive: 7200RPM 500gig seagate with 16 meg of cache
External HFS+ drive: 7200RPM 160gig seagate with 8 meg of cache
Both use the identical dirt cheap firewire/USB2 enclosures I found at the local PC shop
| WRITE (Bytes/Sec) | WRITE (MB/Sec) | Read (Bytes/Sec) | Read (MB/Sec) | |
|---|---|---|---|---|
| HFS+ Firewire | 6049969 | 5.77 | 91697378 | 87.45 |
| NTFS Firewire | 6645725 | 6.34 | 19899810 | 18.98 |
| HFS+ Local | 6565372 | 6.26 | 90154137 | 85.98 |
| NTFS Local | 6495106 | 6.19 | 16776180 | 16.00 |
Conclusions:
HFS+ is obviously doing some kind of caching on those reads, as there's no way you can get 85+MB/sec off a plain old 7200rpm drive, let alone the 5400rpm Local drive in the macbookpro. For Actual Use, I can't tell the difference between the NTFS and HFS+ drives
Also, the read/write speeds suck compared to the 30/25 odd MB/sec windows reports when reading/writing files to the disk. But windows lets you enable write caching for removable drives. Maybe OSX doesn't do this. I don't know.
Apart from that, it keeps up with HFS+ and in some cases beats it.
That's Not Half Bad. I might send some my hard-earned paragon's way.
Tuesday, October 30, 2007
How to manually send an email using Rails' ExceptionNotifier Plugin
We have a situation in our rails app where we want to catch an exception and display a custom error message to the user, BUT we still want the exception notifier to fire, so we know all the detailed backtrace data etc, and can deal with it if it's a problem on our end.
Without Further ado, here is the code.
begin
# b0rk b0rk b0rk
rescue => exception
fake_params = { :id=>some_id, :etc=>'etc' }
fake_request = ActionController::AbstractRequest.new
fake_request.instance_eval do
@env = { 'HTTP_HOST'=>'fake_host' }
@parameters = fake_params
end
ExceptionNotifier.deliver_exception_notification( exception, ActionController::Base.new, fake_request )
Enjoy :-)
Monday, June 04, 2007
5 Things that I don't like about Ruby
I can't remember the quote or source, but there's a pseudo programmer-interview question which goes something like this:
"What's your favourite programming language?"
"OK, what are 5 things that are wrong with it that other languages do better?"
This is something I've thought about from time to time, and so I figure I'll give it a shot. Obviously ruby is my favourite programming language at the moment, mostly(at the moment) due to the map and inject functions :-)
1. Green Threads are Useless!
The ruby interpreter is co-operative - it can't context switch a thread unless that thread happens to call one of a number of ruby methods. This means that as soon as you hit a long-running C library function, your entire ruby process hangs. I encountered this situation, and tried then to ship it out to another process using DRb. This was even more useless, as when you do that, the parent process blocks and waits for the DRb worker process to return from it's remote function... which doesn't happen as the worker is blocking on your C library function :-( I ended up having to create a database table, insert 'jobs' in it, and have a seperate worker which polled the database once a second. STUPID.2. You can't yield from within a define_method, or write a proc which accepts a block
It appears to be to do with the scoping of the block, but in ruby 1.8.X, this code doesn't work: class Foo define_method :bar do |f| yield f end end # This line raises "LocalJumpError: no block given", even though there obviously is a block Foo.new.bar(6){ |x| puts x } The other way to skin this cat is as follows, which also doesn't work :-( class Foo define_method :bar do |f, &block| block.call(f) end end # The "define_method :bar do |f, &block|" gives you # parse error, unexpected tAMPER, expecting '|' # :-( This means there is a certain class of cool dynamic method generating stuff you just can't do, due to stupid syntax issues. Boo :-(3. The standard library is missing a few things
I vote for immediate inclusion of Rails' ActiveSupport sub-project into the rails standard library. I'm sure I won't be alone in thinking this.4. Some of the standard library ruby classes really suck.
Time, I'm looking at you. Strike 1: Not being able to modify the timezone. Seriously, people need to deal with more than just 'local' and 'utc' timezones. Yes I know there are libraries, but they shouldn't need to exist. Timezones are not a new high-tech feature! Strike 2: The methods utc and getutc should be utc! and utc, in keeping with the rest of the language. This alone has caused several nasty and hard-to-spot bugs Strike 3: What the heck is up with the Time class vs the DateTime vs the Date class? This stuff should all be rolled into one and simplified. The Tempfile class is also notably annoying. Why doesn't it just subclass IO like any sane person would expect?5. The RDoc table of contents annoys me
This is probably more "Firefox should have a 'search in current frame'" feature, but under http://ruby-doc.org/core/, have you ever had the page for say Array open, and wanted to jump to say it's hash method? I usually do this using firefox's find-as-you-type, but seriously, try doing just this in the rdoc generated pages with the 3 frames containing everymethodever open. Cry :-(Monday, April 23, 2007
HOWTO: Create a GParted LiveUSB which actually works WITHOUT LINUX
EDIT:
Turns out there is a windows version of syslinux, to be found HERE.
If I'd kept reading for about 2 more minutes I would have found that out and managed to avoid pretty much all of the timewasting I did last night. Sigh. At least the other people trying to make it work using loadlin indicates I can't have been the only one to get it wrong :-(
Also, the graphics card thing is a non-problem. Just chose Mini X-vesa in the gparted boot menus and it's fine
Moral of the story? Just because you've found a solution doesn't mean it's the best one. Keep looking until you can be sure it is!
So, I wanted to repartition my hard drive tonight. I've used GParted before and it was brilliant, so off I went to download the liveCD again.
Once at that site, I saw the LiveUSB option from the left-hand menu, and thought "Brilliant, I don't have to waste a CDR and it will be much quicker anyway!"... Little did I know that PAIN and DESPAIR awaited me. I'll publish how I resolved this in the hope that less other people will have to.
Step 1: Download the GParted LiveUSB distro
I clicked 'Downloads', from the navigation, followed the liveUSB links, and wound up here:http://sourceforge.net/project/showfiles.php?group_id=115843&package_id=195292
I downloaded gparted-liveusb-0.3.1-1.zip, and unzipped it. Hooray, now what?
Problem 1: The GParted LiveUSB documentation is crap!
The GParted LiveUSB information here says firstly I need to download a shell script, then I run it and copy some files to my USB key... Apart from a link to one forum post here that's it. Documentation? Instructions? Why do we need those? What could POSSIBLY go wrong?Problem 2: Running shell scripts on windows doesn't work too well
The above shellscript invokes syslinux, and just about everything else on the net that talks about creating bootable floppies/USB keys also sooner or later invokes syslinux also. This seems to set up the boot record on the USB key so that you can boot linux off it. DOS used to have a utility like this called 'system' or 'sys' or somesuch but I can't remember. Seems simple enough, except I NEED LINUX TO RUN IT. Actually no I don't... see above. oops
In my humble opinion, if I was running linux already, I wouldn't need the liveUSB, I'd just apt-get install gparted and run the damn thing. Yes some travelling sysadmins might have a linux box at home and also need a usb key to take around, but I'm not one of them. The entire reason I'm trying to get this liveUSB to run is because I DON'T have linux.
So, I read that forum post, and noticed at the bottom someone using loadlin to load linux from a DOS system. Aha!
Step 2: A whole crapload of google searching and researching...
As I can't make my USB key linux-bootable without linux, I need to make it DOS-bootable, then get loadlin to load the linux kernel that comes with the gparted liveUSB. I'm going to skip all the boring details as it took me frickin ages and just explain what to do...
Step 2.1: Download a DOS bootdisk so we have DOS
Goto http://www.bootdisk.com/bootdisk.htm and download the "Windows 98 SE Custom, No Ramdrive" boot disk. This gets you an executable which expects to write to your floppy drive... except I don't have a floppy drive. BAH.Step 2.2: Extract the DOS bootdisk image with WinImage
Step 2.3: Make your thumbdrive bootable
Step 2.4: Get loadlin
Step 2.5: Copy files onto your USB key
Step 2.6: Make loadlin run automatically
Note: This is like in the forum post here, except it actually works. I think that's out of date.linux noapic initrd=initrd.gz root=/dev/ram0 init=/linuxrc ramdisk_size=65000
(for those interested, those are the kernel boot parameters which I stole that out of syslinux.cfg from the gparted liveUSB distro. If that file changes, so should your loadlin parameters)loadlin.exe @loadlin.par Step 3: GO GO GO
Reboot your computer! If you've set up your BIOS properly to boot off USB keys, your computer should now boot the GParted liveUSB. HOORAYZ!!!!1111Step 4: cry
That's as far as I got, because the version of X.org on the liveUSB doesn't seem to like my NVidia 7600GT, so I'm stuck with a command prompt. Those of you with other graphics cards however should be fine. Whether the liveDistro includes command line partitioning tools I dunno, I might go look at that now.
If anyone would like to copy/distribute these instructions, or edit copies/etc, you are free to, as I am putting this particular blog post in the public domain under the creative commons public domain license.
Subscribe to:
Posts (Atom)