Monday, January 31, 2005
Tool for Thought - Steven Johnson writes in the NYT
Steven Johnson had an essay in yesterdays NYT entitled "Tool for Thought" in which he writes about his use of an collection of his own writings, together with selected passages from books he's read (entered by hand, with the help of a research assistant!) , which togther serve as "an archive of all my old ideas, and the ideas that have influenced me." By searching this, he's certainly able to find old information he knew was there, but it also serves to find information he didn't know he was looking for!
The software he uses to access this archive is DEVONthink (a Mac only product), which is able to make connections between articles based on content, so it is able to suggest other articles, even without there being direct matching keywords in those articles. As a writer, both of articles and books, this chaining together of the archive fragments helps Steve very much in coming up with ideas, finding connections that would not have occured to him in the first place.
Steve goes into a lot more technical detail on the process in his own blog.
Since the data was assembled by Steve in the first place, the data is focused in his direction, and the results are very relevant.
Steve also makes the very significant point that the fragments and articles are also generally quite small - of the order of up to 500 words, and this seems to be a very good size for these connections to come out. On the web, many pages are around this sort of size (or at least the useful information once all the navigational markup has been removed is around this size), but moving to the "conventional" desktop search that Google et al have released recently, we are suddenly searching potentially huge locally produced documents - and the search breaks down at this point. (Even though the better desktop search tools are able to give a preview of their results showing the context around the matching word, I know of no tools that allow you to say that you want the best matching paragraphs within the document, rather than simply the paragraphs that match in the document which is somehow ranked highly across the whole document).
For Windows users, the comments on Steve's blog suggest that askSam (which describes itself as a free form database) may be worth investigating as an alternative to DEVONthink. Other programs that perhaps have some similarity include
The software he uses to access this archive is DEVONthink (a Mac only product), which is able to make connections between articles based on content, so it is able to suggest other articles, even without there being direct matching keywords in those articles. As a writer, both of articles and books, this chaining together of the archive fragments helps Steve very much in coming up with ideas, finding connections that would not have occured to him in the first place.
Steve goes into a lot more technical detail on the process in his own blog.
Since the data was assembled by Steve in the first place, the data is focused in his direction, and the results are very relevant.
Steve also makes the very significant point that the fragments and articles are also generally quite small - of the order of up to 500 words, and this seems to be a very good size for these connections to come out. On the web, many pages are around this sort of size (or at least the useful information once all the navigational markup has been removed is around this size), but moving to the "conventional" desktop search that Google et al have released recently, we are suddenly searching potentially huge locally produced documents - and the search breaks down at this point. (Even though the better desktop search tools are able to give a preview of their results showing the context around the matching word, I know of no tools that allow you to say that you want the best matching paragraphs within the document, rather than simply the paragraphs that match in the document which is somehow ranked highly across the whole document).
For Windows users, the comments on Steve's blog suggest that askSam (which describes itself as a free form database) may be worth investigating as an alternative to DEVONthink. Other programs that perhaps have some similarity include
- Info Select (a Personal Information Manager)
- Zoot (an "information processor")
- Onfolio
- Ultra Recall
# posted by zmarties @ 1/31/2005 07:21:00 pm 
