Visualize This Isbn:978-0-470-9448-2 © 2011 Nathan Yau, Published by: Wiley Publishing, 10475 Crosspoint Boulevard, Indianapolis, IN 46256 Author: Nathan Yau, cover Price $39.99, 358 pages
Returning from a trip, I was very pleased to see a review copy of Visualize This waiting for me. The author, Nathan Yau has a great web site, FlowingData which I have been following for some time. I enjoy the site a lot and consider it to be one of the best sites for data display and inspiration.
This book is not for beginners who are not interested in getting their hands dirty with code and cleaning data. If however, you have done the basics, and maybe a bit more than the basics with Excel and want to explore the next level up, then this book is for you.
Visualize This is quite different from many of the other books that I have read on visualization in a good way. The author comprehensively explains how he approaches the task of coming up with useful concepts for data display and then shows you how to do it yourself. He immediately points out that "basic" data display can, and most often is, done using Microsoft Excel. Happily, we have little of that here. The tools of choice are Python (a popular programming language available on all platforms), and R (a sophisticated open source and free data statistics package), coupled with Adobe Illustrator, (a fairly expensive product from Adobe). He also highlights the use of Google Refine, a fairly new tool from Google for cleaning up messy data. This is probably the first time I have seen this useful tool mentioned in print, as opposed to seeing it on the web. The author does mention a number of commercial and open source alternative tools but the bulk of the step by step examples use, Python R and Illustrator.
I was very happy to see the concentration on these specific tools as I have been learning Python and R for a while now, and have Illustrator available (and have used it a bit) as well as the free alternative, Inkscape, which the author also references.
Throughout the book there is a useful survey of other tools, both open source and commercial for obtaining, formatting, and displaying data. There are a number that I have not yet explored. So this book serves as an excellent resource.
Inclusions of specific instructional material with real data and readily available tools sets this book apart.
The book is clearly the product of someone who has thought deeply about the process of producing useful visualizations. The chapter organization echo the suggested processes:
- Chapter 1 — Telling Stories with Data
- Chapter 2 — Handling Data
- Chapter 3 — Choosing Tools to Visualize Data
- Chapter 4 — Visualizing Patterns over Time
- Chapter 5 — Visualizing Proportions
- Chapter 6 — Visualizing Relationships
- Chapter 7 — Spotting Differences
- Chapter 8 — Visualizing Spatial Relationships
- Chapter 9 — Designing with a Purpose
In short, the book belongs on the bookshelf of anyone who is interested in the subject of data visualization beyond the basic level. It has a permanent spot on my reference shelf.
An acquaintance and I were discussing what to do about moving from an older PHP based CMS, used in a somewhat complex University setting, that had grown unwieldy, to something newer like Drupal. Here are some thoughts.
I believe that if you are going to make such a major shift it is worth it to get out of any PHP framework. PHP while very popular and having many large sites is an inherently clumsy approach to programming, and therefore tends to introduce problems into most if not all the products derived from it.
Keep in mind that all of the below are Django projects therefore share a lot of common code and building blocks, making for a very rich ecco system. That is you can often take features from one and ad them to another.
There are a huge number of very large and high traffic sites implemented in Django
The main Django site is :http://www.djangoproject.com/
See this for some perspective:
That being said there is no real reason to start off with "naked" Django. There are numerous CMS approaches all available as Open source, and by the way the Django/Python world is generally BSD licenced (one of the most liberal licences)
Projects worth a look at are (in no particular order):
Fairly basic and new but clean design:
Comparison: Django-Pyramid vs Rails
Pinax: A prefab set of Django components:
For an overwhelming overview check out this comparison site
Mostly they go to a Web agency of some sort then go along with what they say. They pick the agency based on prior work, which may or may not really be analogous to their specific needs. Usually the choice is made on what prior web sites look like, so a high premium is placed on design, often over functionality.
Some Web providers make, shall we say, interesting claims, when it come to Search Engine Optimization (SEO). This a rather large topic that won't be covered here, but be very wary of exaggerated claims. Google is pretty smart in this area and there are increasing stories of sites and companies loosing SEO rankings due to bad practices.
Some considerations for choosing Web hosting tools and suppliers follows.
It is interesting how little most people understand about Web sites. Most Web sites are accomplished using some sort of Web Framework, such as Wordpress or ASP.NET MVC, Microsoft technologies. Some of the other top frameworks are Drupal (used by WhiteHouse.gov), Joomla (examples), Ruby on Rails (examples) and my personal favorite Django (used by National Geographic, PBS, and NASA, others ). There are literally hundreds if not thousands of such frameworks each with its own strengths and weaknesses. The framework often determines what the Web provider can do in a timely and cost effective manner and how much the end user can do for themselves. There is no one size fits all solution.
Beyond flexibility and efficiency, security is another aspect of Web Frameworks. It is not simply a matter of getting the firewall right. With some notable exceptions, firewall configuration is now pretty well understood, Only open port 80. But sadly that does not provide much security for many of the most common web attack vectors. Specifically it provides no protection for Cross-Site Request Forgery (CSRF) errors nor XSS issues explored here XSS Prevention Cheatsheet nor does it provide much for mis-configured servers and databases, the most common security issues for Wordpress (details on Wordpress security advice is here), and other frameworks. In fact this list of attacks http://www.owasp.org/index.php/Category:Attack contains none that are inhibited by firewall configuration.
Most any Web framework provides a fairly large exposed surface from a security standpoint.
Another consideration is where the site will be hosted. Many agencies do try to get the hosting business at the same time as the development business. Sometimes there is an urge to host the site on some sort of internal capacity. In general one would be much better of having the site hosted by a real Web host rather than a small service provider for security, scalability and reliability, not to mention that most large Web hosts have inherent disaster backup that local companies would have a hard time replicating. Security threats such as denial of service (DOS) attacks are best handled by the larger providers.
There is however an emerging class of frameworks that is pretty bullet-proof, known as static sites. That does not mean that the sites do not have controlled levels of interaction or animation, but rather that the site and framework has no executing code on the server, it is in a very real sense entirely pre-compiled. Many smaller company sites are very amenable to this approach.
A leading implementation is Blogofile. Interestingly this was just recently implemented on Google App-engine, (Details here) It can also run on a free micro site instance at Amazon EC2 and or EC3 for very little. So for medium sized traffic the hosting can be free or nearly so. Of course these static sites are easy to host on internal infrastructure as well. Static sites do not however provide for user input, other than basic site search or commenting, usually provided by a third party service such as DISQUS.
Static sites are maintained off line, behind the firewall and once updated the entire site, or just the "diff" (an automatically calculated change set) is pushed to the server for new articles and content, most ideally from a change control system such as GIT or Mercurial. This provides excellent change tracking, control and distributed contribution and multiple author capability.
It is further, easy to incorporate differing media types and embed links such as pictures video files and PDF's.
Do yourself a favor and before you invest or get invested in a full framework give static sites a look. Further, if your needs are more modest things like Google sites and other hosted full solutions like Squarespace can be quite satisfactory.
This book however chooses to explore all the platforms and manufacturer specific approaches. It may be the only book to attempt that very hard task. The speed with which the mobile platforms have evolved makes any print book almost hopelessly out of date upon publication. Indeed due to this fact the book only has two paragraph references to the new Windows mobile phone system.
The book has several useful chapters on mobile phone system evolution, which helps to understand where each operating environment resides and how it got there. This in turn is quite useful in understanding how likely your programming product can be implemented across specific platforms.
The book does a good job of explaining how designing for smaller screen sizes is really different. It also has a good section on design patterns to support the smaller screen sizes.
Where the book falls somewhat for me is in the lack of practical full examples. According to the author this is somewhat by design. I found it quite disturbing however. The various features are well explored but, implementation is really up to the reader to figure out. This can be quite exasperating. This is so because the various platforms each have pretty unique capabilities and restrictions and enhancements. Finding common implementation patterns that will work across a large section of platforms is my goal and I was left somewhat disappointed.
All in all an impressive resource. Useful addition to the bookshelf for the experienced programmer wanting to get up to speed on the Mobile Web.
This has been a Google Christmas for us. It began with the unnannounced arrival of a CR-48 laptop, which has been quite a pleasure to use (see the prior post).
My daughter and I both had the original G-1 phone, and they were on their last legs. Batteries that could barely last a few hours and then only if you did not get a phone call! We were seriously overdue for new phones, and the rumors of the Nexus S convinced us to hold out until this "super-phone" was available.
I am seriously impressed with how far the Android system has come. Even though, this review is still "early" impressions; it takes a while of day-to-day use to completely understand the strengths and weaknesses of any device.
The speed, form factor and screen readability (particularly outside in the sun) are nothing short of outstanding. Battery life is outstanding. Since everyone's usage is so variable , I can't be too specific on battery life, but both of us have been using the phones extensively all day mainly for surfing without running into a need to recharge. By the way the connector is a standard "micro" usb, which Motorola an Kindle both use...
The new and improved applications are welcome to those of us who were frozen on Android 1.6. Some of the new apps and features are discussed below.
The first of two outstanding features that I'm still adjusting to is Swype, a great alternative keyboard entry capability. Secondly, the phone has outstanding speech recognition. In fact, I used the speech recognition to find and install most of my applications and it was seamless. I also used it to send a text reminder which means that you say something like remind me to check the oil this week and you get a nice e-mail with the message nicely typed out in your inbox, it works great.
I'm also enjoying the e-mail notifications from multiple accounts. I use the "mail" tone on the phone to remind me to check mail on the computer since the phone seems to get them sooner than my computers do and from all accounts. The phone handles Google Apps accounts and regular Gmail accounts very well. I was impressed with the automatic integration of multiple address books into contacts. I took the option of only having contacts with phone numbers on the phone. This avoids having e-mail only contacts appearing in your contacts folder on the phone, many of these may be E-mail lists and the like. I don't deal with any Microsoft Exchange or POP3 accounts anymore (thank goodness) but the reports are that those are also handled way more gracefully than on earlier Android phones.
Maps are greatly improved but I have not had much occasion to use them for real navigation.
Call quality, which also features noise reduction, is excellent. The noise reduction uses a rear facing microphone to detect and eliminate background noise. Call voice quality seems quite clear and good.
|From Drop Box|
This is a picture of a desert my Wife made for one of the Christmas gatherings "Bouch de Noel"
The regular camera takes very good pictures and has auto white balance and some additional controls. I'm actually a fairly experienced photographer (used to do it professionally) and have been surprised and impressed with the picture quality. However, as I mentioned previously, it's still pretty early to say for sure. I have seen some complaining about the camera but my experience would lead me to think it may be an operator error! I was also impressed with how easy it is to share the images with multiple services; the integration with Picasa is outstanding (the "Gallery" app automatically syncs to Picasa). The front facing camera is clearly intended for video chats but at the moment Google Video is not available. The forums say that Tango, another package may be an option. I think I'll wait a bit though.
In many ways the phone seems way more advanced than the CR-48. It installs a wide variety of applications and files (like music and multimedia files) readily while the CR-48 so far deals exclusively in the cloud.
I have yet to experiment with the phone's tethering capabilities, but I'm definitely looking forward to playing around with it. There appears to be some confusion about T-Mobile's attitude to it, but I'm sure they'll figure it out.
Voice over IP is another challenge I have not quite gotten the hang of yet. The T-Mobile Unlicensed Mobile Access (UMA), a technology that allows mobile users to send and received phone calls via Wi-Fi, is not yet available, (don't know if it will be). Alternative VOIP approaches such as Sip services which seem to be a bit fiddly. I was able to cobble a solution together (based on Google Voice and Sipgate) but it only worked in one direction, they could hear me but not vice-versa. The help areas suggest that I should update my router firmware, but not sure I want to take that leap yet. I have downloaded updates but it remains a task for another day. Others report success with this path, so maybe I'll get around to it before too long. Google Voice seems to work fine on its own. I'm still having some difficulty getting the full use case though. I did allow Google Voice to take over from the carrier voicemail and I will say that I enjoy the voice transcriptions; they actually are useful.
At the time I was getting going with the phone the Delicious fiasco was also going on and I began to switch to Diigo for bookmarks and I discovered they have a suite of cool apps that integrate well with Android. Power Notes (part of Diigo) allows access to all bookmarks and notes, and is somewhat more lightweight than other apps. "Chrome to Phone plus" is also available (from Diigo) and allows sending "read later" items and maps to the phone. I could not get the Google version of the Chrome to Phone working, but it seems the Diigo product is far more functional anyway so I don't really mind.
The new Google Reader app is a nice addition, and provides a much more seamless user experience than the previous version.
I'm looking forward to a better VOIP experience (and maybe UMA) (both via wifi and maybe even 3g), and Video calling via Google chat, I'm sure we are just at the beginning of the new features we will be seeing rolling out on the platform.
The new Car Home view is also a very nice addition for on-the-road. It provides much larger buttons for the main functions you might use while driving, including handy large buttons for calling frequent contacts. It sort of looks like the Windows Phone 7 interface!
(added by my daughter) >> OVERALL, I LOVE THE PHONE AND CAN'T PUT IT DOWN. MY FAMILY HAS TO SURGICALLY REMOVE IT FROM MY HAND WHEN IT IS TIME TO SIT DOWN TO DINNER. TOM BRANDER, PROPERTY OF GOOGLE. (it actually is a fair summary of my feelings.)
I got one.. That is the most amazing thing!! The new CR-48 from Google is a nice compromise between a netbook and a full size laptop. For instance I'm typing this laying down on the sofa something I would not comfortably do with one of my laptops.
While I don't really mean it to be.. I can't help but make this a bit of I got one and you did not post,, sorry just can't help it..
Damm this browser based blog editor just erased a bunch of text when all I was trying to do was correct some spelling and it seems to lack an undo function. The answer for this seems to be to manually save a bit more frequently.. (I'm using the Scribefire Chrome app to compose this post)
What I was saying, before I lost it, was that it is hard even for me to completely leave behind all client based apps except for those available in the Chrome ecco system. Now I have been largely committing my life to Google and the cloud.. I use G-mail and Google apps extensively, I was one of the 1st to get an Android phone (the original G-1) which is still in use although admittedly on its last legs... I plan to get the new Nexus S on the 1st day available (12/16/2010) this week but due to the way Best Buy is handling it I must order on line (to get the T-mobile upgrade price) which is a bit of a problem.
One thing I'm discovering is that this CR-48 little machine is faster than my Windows Vista 64 bit 4gig ram dual core laptop for simple web browsing.. I have the two next to each other and it is quite evident. My wife also observes that the screen seems way sharper and more readable than the Windows machine...
I tried to use the Chrome TV app but all those stations require Windows media player.. so no go for the time being..
Pandora works fine.
IRC (Internet Relay Chat) is problematic, the Mibbit app is banned from freenode (which is the IRC host of choice) and freenode has a web app but pretty primitive..
Last night a small group of us here in Birmingham, I think we have identified 4 lucky recipients, decided to form a CR-48 users group, meeting to be announced. The Group is here https://groups.google.com/forum/#forum/bham-google-cr-48-group . Of course this was largely organized via twitter!
I've read a lot of bitching about the trackpad but I have found it to be a minor adjustment factor. It is working fine for me,, just need to figure it out a bit better...
One friend has immediately turned the dev mode switch on to gain access to the underlying system.. I plan to hold off on that to more force myself to see how to live as Google intends using the stock install and browser based apps...
With both my Widows machines and Linux I have routinely disabled sleep mode,, I use the temporary screen off.. but not the processor because it seems to me that the odds of the machine ever coming back is fairly low and it also takes an inordinate period of time.. Happily this is not a problem on the CR-48 it just flat handles it perfectly and very fast,, A slight nig is that it does seem to take the wireless (wifi) net connection a few extra seconds to wake up so things don't get going 'till that process finishes.
Just realized that the alt/backspace works like delete key which got the heave ho. Ctrl-next window is screenshot and ctrl-o (that is small 'o') to view it.
Battery life seems fantastic but will take a while to fully assess.
ctrl-alt-t opens a terminal.
ctrl-alt-/ gives a handy keyboard function guide,, there are a ton of modal shortcuts,, that will take a while to learn....
for those interested in developer mode http://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/cr-48-chrome-notebook-developer-information
I'll probably modify this a bit as I have some time, and add the obligatory picture.
And did I say this well enough, Thank You Google!!!
- Version Control with GIT.
- Learning VIM (a very powerful text editor) Along with Command line orientation.
Many do not understand that the Web Browser is now a full fledged operating system. It has I/O handling, for the network and local storage. It supports the machine interface across many operating systems and machine types and it handles the presentation of data along with user interaction. It is fair to say the the browser is now the most cross platform operating system in existence today.
Web server based programming requires so much learning across so many domains to grasp, that I put the whole subject of HTML/CSS/JS at the bottom of my very long list of learning tasks. I got the basics but I figured that I could get away with just the basics, after all it was "just presentation". Well presentation counts!
On twitter someone (and I've tried to remember who so I can attribute) made the observation that if you do not know the layers of the programming model you are very likely to be working in the wrong one. Similar aims can be achieved in the database layer the layer just above the Database known as the ORM or even higher in the view or template layers or right in the browser using Javasript (this layer description is somewhat simplified).
The concept is really heating up as the mobile Web gains market. With new standards, since most mobile browsers use the same core (Webkit) very rich mobile apps can be almost entirely coded "in browser".
I avoided three things early on which I wish I had just gone ahead with at the time.
- Version Control with GIT
- Learning VIM (a very powerful text editor) Along with command line orientation.
VIM is known as a very sophisticated and complicated text editor for programmers. It's fearsome reputation for having a long learning curve led me to look for any other simple tool. Having been away from coding since the days when I used a 1200 baud hardcopy terminal, (a huge and welcome improvement from the 80 Column punch cards) except for some VB on Windows, I naturally began my search for Integrated Development tools that would do my code for me (like Visual Studio). Eclipse is a comprehensive Integrated Development Environment (IDE) similar in scale and features to Visual studio, but Open Source and free from IBM. It is a great tool with many great features, but it has its' own steep learning curve. Google App-engine which was one of my early explorations, requires Eclipse and I use an open source report writer Birt which is based on Eclipse, further, the Google development environment for Android is Eclipse based. So Eclipse is an important tool to have in the toolbox. Pydev, its' Python tool set is one of the best available.
Sometimes you just don't want to fire up Eclipse to change a few lines in a file or if you are doing editing on a server you may not have Eclipse available. It is really important to have knowledge of a good text editor that will work across all platforms.
I First thought of Notepad then Wordpad then Notepad ++ (an excellent product). At Pycon (the Interanational Python gathering) I realized that all the cool kids were using VIM (ok maybe a few were using Textmate or Emacs) but no one was using Notepad++, why? Well VIM is quicker and its earlier cousin VI is installed by default on virtually all systems (save windows). VIM has a great Windows version but it must be downloaded and installed.
Vim is a mode based editor. This means that it has keyboard commands for most editing operations. The recommendation is that you never use the mouse, and you never or rarely use insert mode, which makes Vim behave like those less efficient editors you are more used to. VIM feels really clunky and foreign at first. If you stick with it you will soon be going "oh wow that is cool!" and you too can become one of the cool kids! As well you will feel more confident every time you face the Terminal command line on the server. Not to mention that you will gain a better and better understanding of differing file formats and all the "helpful" invisible characters that Microsoft products hide in files, which in turn cause what IBM used to call "unpredictable results". I wish I had just started learning VIM earlier, it has so many features for coding, it is just shy of being a full Integrated development environment (IDE), and it has the benefit of being everywhere, so it is about the closest thing you can find to a universal tool.
My purpose here is just to give the recommendation for VIM, smart people have produced some great resources for learning VIM. Try these for starters.
Derek Wyatt has a very good set of Videos including this intro to VIM
Bram Moolenaar The originator of Vim has a very good intro
NET-Tuts put this list of 25 resources together
The Open Source Development world is a big one. First, I thought I'd have to convert everything and all of my machines to Linux. In fact, unlike vendor provided solutions, (read Visual Studio and components here) The open source arena provides what seems like too many choices. So it becomes important to pick what you pay attention to or you will waste a lot of time just investigating alternatives.
Git web Site
I avoided three things early on which I wish I had just gone ahead with at the time.
- Version Control with GIT
- Learning VIM (a very powerful text editor) Along with command line orientation.
I got some brief exposure to GIT as a version control system when putting up my 1st application on Google App-Engine. I thought it was horribly complex.I'll confess to using Tortoise GIT (a GUI interface) to help my learning process.
My old school learning had taught me to keep separate folders and to be meticulous about backups. I began using Subversion (another version control system) on my local machine, and I got into more Python I explored Mercurial. which seems somewhat simpler than Git and has the advantage of being Python based.
Even in the two years I have working in the Open Source world the version control tools are much better. GIT now runs well on all platforms (including Windows). It seems clear that GIT is the way to go, not that the others are not good but the network effect of momentum that GIT has achieved will be very hard to slow down. GIThub has done a remarkable job (now hosting over a million projects) of providing a continually enriched feature set for software development and collaboration to the point that its is sucking the air out of the room for competing Revision control systems.
Recently BitBucket the leading public Mercurial repository (now hosting about 60,000 projects) has been taken over by Atlassian , which may be able to inject more life into the project. Increasingly the version control systems are able to interoperate so you can use GIT to pull and push to and from Subversion and Mercurial.
I can't say that I have completely worked GIT into my daily workflow, but it is high on my list of things to do. when I need version control for collaboration GIT is the go-to tool I'll be using. You can see my review of Pragmatic Version Control Using GIT on Dzone . GIT is just not that complicated (although it can do some pretty advanced stuff you don't have to). Just get on with it and get going with GIT!
HTML5, Up and Running Isbn:978-0-596-80602-6 © 2010 Google Press, O'Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopal, CA. 95472 Author: Mark Pilgrim Cover Price Us $29.99 205 pages
I recieved a free copy of this book from O'Reilly/Mark Pilgrim to review as part of the D-Zone review team.
Mark Pilgrim is a developer advocate at Google. His web site is http://diveintomark.org .
Although HTML5/CSS3 are very new, one of my interests is in creating mobile web-apps. It has become quite clear that Mobile Web will be a superior development environment going forward for the vast bulk of applications needs. A major advantage of web-apps contrasted with native-appscourse that they are really the write once run on any device solution. Coincidentally, since most Mobile browsers are Web-kit based, there is a fair amount of HTML5 goodness and consistency in that environment.
I was extremely impressed by this book. At first I was a bit puzzled by the long, but very well written coverage of the history of HTML standards. Mark uses this history to cleverly present the evolution of the standards themselves. This explains why in many cases the new standards can be employed today with little problem of reverse compatibility with older browsers.
Some reviewers have been critical of the book for lack of depth or too much history. I disagree and I found the history helpful. History provides the all important context for why things work the way they do. As far as depth, some browsers and or features are simply specs or tags without differential implementation as. This prevents some elements from being discribed in as much depth as some might like but it represents the real world. It is not a fault of the book, but rather the fault of the current state of the specification. Some areas of the HTML5 spec are simply left for implementation decisions, and do not have "standards" declared. The book does a good job of defining the current state, but this will rapidly become out of date as browser updates are released. Naturally, we will need to check the web for the up-to-date info.
Each HTML5 enhancement is presented with extensive instructions for what can be done to support backwards compatibility. The chapters are as follows with my notations of the topics covered.
- How did we get here? The historical bacground provides great understanding for immediate use and backwards compatibility.
- Detecting HTML5 Features; best practices for testing browser support and in some cases providing it.
- What Does It All Mean? An example page of markup enhancement, which maintains backward compatibility.
- Let’s Call It a Draw(ing Surface) Canvas and What You can do with It
- Video on the Web: A good overview of the current standards and issues and what to do for the future.
- You Are Here (And So Is Everybody Else); Geolocation.
- The Past, Present, and Future of Local Storage: Overview of the past and future of Browser storage
- Let’s Take This Offline: How to plan for off-line usage of your web-app
- A Form of Madness: New forms tags and capabilities.
- “Distributed,” “Extensibility,” And Other Fancy Words: Interesting discussion of microformats.
- The All-In-One Almost-Alphabetical Guide to Detecting Everything
Mark can also be found on Twitter @diveintomark