Check for the last state. Something like this:

Keep a variable, say, last_scroll_position, and when you have a scroll, if last_scroll_position - current_position > 0, the user scrolled up, and down if it's less than 0.

<script type='text/javascript'>
$(function(){

  var CurrentScroll = 0;
  $(window).scroll(function(event){

      var NextScroll = $(this).scrollTop();

      if (NextScroll > CurrentScroll){
         //write the codes related to down-ward scrolling here
      }
      else {
         //write the codes related to upward-scrolling here
      }

      CurrentScroll = NextScroll;  //Updates current scroll position
  });
});

To differentiate between scroll up/down in jQuery, you could use:

var mousewheelevt = (/Firefox/i.test(navigator.userAgent)) ? "DOMMouseScroll" : "mousewheel" //FF doesn't recognize mousewheel as of FF3.x
$('#yourDiv').bind(mousewheelevt, function(e){

    var evt = window.event || e //equalize event object     
    evt = evt.originalEvent ? evt.originalEvent : evt; //convert to originalEvent if possible               
    var delta = evt.detail ? evt.detail*(-40) : evt.wheelDelta //check for detail first, because it is used by Opera and FF

    if(delta > 0) {
        //scroll up
    }
    else{
        //scroll down
    }   
});

This method also works in divs that have overflow:hidden.

I successfully tested it in FireFox, IE and Chrome.