Postcards from the Fridge

Monday, August 21, 2006

Thanks for All the Fish

Well, today SOC draws to a close, so I decided to write up how things currently stand. Thanks to all the Gnomers for letting me work on this!

Abstract

The point of this project was to investigate why nautilus's startup performance was slow, and to find different ways to fix the problem.

Nautilus's startup is slow because it keeps blocking and allows other applications to run.

Sometimes it is lucky and completes its startup quickly. Other times it isn't, and gnome-panel (and all of the plugins) or other startup applications get in the way.

Problem definition
  1. minimize the amount of time between nautilus startup and mapping of the icons.
  2. Remove the variability in the gnome startup times.
I would argue that it is most important for the file manager to start up. In particular the problem that I was solving (which is one that I see every time I start up any modern based Linux, be it on Fedora, Ubuntu or SUSE). The X session starts, the panel shows up but it takes seconds for the icons to finally appear on the screen.

Tools

The general problem

When I started this project, I had very little idea about the overall architecture of gnome. When do various components start up? Which pieces were responsible for which parts of the system? There is documentation for some of the individual components (Nautilus's documentation is pretty good), but nothing that describes the overall startup of the gnome desktop. There are quite a few articles describing the exact startup sequences of Linux from the boot loader to the init scripts. It would be helpful if there was something similar for gnome, so it was at least clear how things were intended to work.

The problem of understanding and analyzing the complex startup of gnome is actually a more general problem. Gnome starts multiple threads, and multiple processes, each with interlocking dependencies. A few general tools have been developed for Linux which help to understand these startup sequences. No single tool seems to have all of the features needed to truly understand what is happening, so I tried a bunch of different tools.

1) Bootchart (http://www.bootchart.org/)

The first tool that I used to determine how exactly things were starting up was the "bootchart" tool. This tool was originally designed to track the startup of programs in the Linux boot process. Fortunately it is also flexible enough to monitor and track any other set of processes that are started at a later date. It gives a nice graphical chart of when various processes started, and how much each used the cpu during the specified time period.

The initial bootchart images were the first indication that something was unusual about nautilus startup. Different charts of the same gnome startup sequence provided a radically different view when things started up, and how longs things took to start. Sometimes the panel would begin first, and sometimes the gnome-settings-daemon would. This was my first clue that some sort of race was happening. If there wasn't a race, each startup should have an identical series of steps, with little to no variability.

2) strace + 'access' hacks and Federico's visualization tool.

Next I started to use the analysis tools and techniques that others had used before me. In particular, I used strace to generate, and Federico's tool to analyze the result. This provided the most complete data. I've detailed in an earlier blog entry how to get it all started up.

Once thing I haven't mentioned yet, is that I wrote a small python script to sort through the strace results. This will replace "pids" with the actual name of the processes. This makes reading the strace results much easier. (It will trace clones, forks and execs, so you can see exactly what is running when...)

#!/usr/bin/python
import fileinput
import string

pid_to_name={}

thread=1

for line in fileinput.input():
__pid = line.split()[0]

__if pid_to_name.has_key(pid):
____start =line.find(" ")
____print pid_to_name[pid] + " " +line[start:-1]
__ else:
____print line[:-1]

__if "execve(" in line:
____param = line.split()[2]
____pid_to_name[pid] = param.split("\"")[1]

__if "clone(" in line:
____new_name = pid_to_name[pid] + (".%d"%thread)
____thread = thread + 1
____new_pid = line.split("=")[-1].strip()
____pid_to_name[new_pid] = new_name

__ if "<... clone resumed>" in line:
____new_name = pid_to_name[pid] + (".%d"%thread)
____thread = thread + 1
____new_pid = line.split("=")[-1].strip()
____pid_to_name[new_pid] = new_name

3) NPTL Trace Tool (http://nptltracetool.sourceforge.net/usm_example.php)

Next, I wanted to try to visualized how the different thread interacted with the hope that I would be able to see some clear patterns as to why nautilus was continually yielding the CPU.

The NPTL trace tool looked like promising avenue, because it had a graphical interface (using ntt_paje), and it was able to analyze a the complex interaction of threaded applications. It can provide a visualization, as well as statistics about the amount of locks held, and report on the various pthread calls that were made.

The trace tool produced the results into a format the Paje visualization tool could open and analyze. Paje itself is a little clunky and requires that the GNUstep environment is installed. I was able to capture a trace of a generic application, but unfortunately when I tried to run the trace tool an nautilus, it seg faulted.

4) frysk (http://sourceware.org/frysk/)

This was another tool that promised to make visualization and debugging of complicated threading applications/environments easy to do. It, unfortunately, is still immature. When I tried to monitor system calls that nautilus made, it also crashed. However, this project is one to watch. It provides a graphical interface analyze how, and allows you to add arbitrary rules to analyze how a process and set of threads were executing.

5) Ideal tool

The ideal tool would be one that shows both when applications are running, and why they vacate the CPU when they are finished. Strace is really a poor man's version of this tool. However, it doesn't have any way to indicate when the scheduler decided to swap the running process, so if an application spends all of its time in user space, the strace method will never detect that the running application has changed.

6) Dishonorable mention


Two tools that I tried to use, but were unsuccessful were AMD's code analyst and Intel's VTune.

AMD get props for releasing their Code Analyist as GPL. It provides a GUI interface to oprofile, and some other niceties. Unfortunately, the number of distributions that it supports is rather small, and it wouldn't build on my Fedora 5 box. Even worse (in my opinion) was Intel's vtune. They provided a binary which could be used on Linux for non-commercial profiling. However, once it detected that my processor was an AMD chip, it wouldn't let me use it.

Experiments: Summary

Since it may have been difficult to follow what happened over the course of the summer, I'm going to summarize my experiments, and show how I came to the conclusions that I did.

Setup

First, my goal was to work with "warm-booting" nautilus, after everything had been loaded into the disk cache. Second, I automated the startup and collection, so that I could run the same test repeatedly, and then analyze the results by either looking at visualizations, or running the tests through gnumeric.

Goals
My goals were to remove the variability in the nautilus startup, and minimize the overall nautilus startup time.

Test 1 Full startup

I added tracing information which started at the first line of "main" in nautilus, and stopped when the icons were painted on the screen. (in icon_container expose_event).

Result:

The startup times were highly variable, and other processes (such as those launched by gnome-panel) would run in between the startup of nautilus and the drawing of the icons.

Test 2 Full startup w/a high gnome-session priority

I edited the ".gnome/gnome-session" file, and set the priority of nautilus to 10, which was higher than that of gnome-panel.

Result:
No change. Nautilus was STILL interrupted by gnome-panel processes.

Test 3 Launch simply (no wm, no gnome-panel)

Next, I removed the "wm" and "gnome-panel" from the session file, and retimed nautilus's startup.

Result:

The nautilus startup times were still highly variable. Nautilus was being interrupted by the gnome-volume-manager, and the various daemons that were launched internally by gnome-session. (gnome-keyring)

One interesting part of this investigation was that sometimes "gnome_vfs_get_volume_monitor()" took and long time to start, and other times it was very quick. Federico had seen this, and he explained what was happening. Various gnome-daemons/services are launched on demand, and this call was causing this to take place.

Test 3 Launch simply (no wm, no gnome-panel) w/a gnome-vfs preloader
I wanted to see if nautilus startup times would because a little, if the gnome-vfs-daemon was prestarted before nautilus started to launch. I modified the disk mount applet to call the gnome-volume-manager, and
exit. Then, I modified the gnome-session-manager to call this "gnome-vfs" preloader, and wait for 2 seconds while it started. The 2 second delay is an unacceptable "real" solution, but it was enough to see prestarting the daemon would increase startup time.

Result:

The variability totally disappeared, and the startup times dropped to about 1/2 second.

Test 4 Launch complete session w/preloader.


I decided to go for broke. I had the preloader within gnome-session, so I decided to try with gnome-panel and the gnome-wm turned back on.

Result:

unfortunately, the variability was back. Upon closer investigation the applications were still interrupting nautilus between the time it was initially launched and the first icons were painted on the screen.

Nautilus was yielding the cpu, and other applications (gnome-panel) were running before the nautilus startup could complete.

Test 5 Launch complete session w/preloader & niced gnome-panel.
Now, I wanted to see what happened if I made the nautilus the highest priority process, and basically made sure that it was the highest priority. However, since I was not able to easily boost its priority (without becomes root), I decided to drop the priority of the processes competing for the CPU. I ran the gnome-panel with a nice of 19.

Result:

Nautilus's startup was very snappy, but the gnome-panel never seemed to be painted. The variability was there because other applications (gnome-keyring) could still occasionally interrupt the startup.

Next Steps

To really solve the startup problem, nautilus probably has to be rearchitected to stop blocking or it must somehow be made a higher priority than the other applications that startup.

Here are some ideas on how to fix things:
0) Incorporate gnome-vfs startup in gnome-session

This should probably be done even if it doesn't completely solve the nautilus startup problem. gnome-vfs is needed by the gnome desktop. Launching it "on-demand" only causes blocking and unnecessary
contention. (I haven't tried this yet. )

1) Rearchitecture Nautilus to remove all of the yielding.

This is a bigger change than I could do. Even minimizing the number of times that it yields would reduce the chance that it would be interrupted.

2) Remove the on-demand loading (or at least short circuit it, so it is never used in a properly running desktop. )

IMHO, the on-demand loading is a bad idea. Ideally all of the dependencies for an application would be running before it starts. It would reduce the amount of times that applications block, and could
possibly reduce thrashing as multiple applications wait for a component to start.

3) Modify nautilus to give gnome-session a thumbs up ONLY after the icons have painted.


Right now, gnome-session and the applications that it launch have a handshake that wait for an application to start. This response handshake would have to be modified to only respond after the icons have been painted.

I've been working on getting this to work, but I can't figured out where the handshake takes place.

4) Boost nautilus's priority

If we can boost nautilus's priority, it may (possibly) not be interrupted. I couldn't figure out how to do this without root privileges.

Summary

In any event, this has been a lot of fun. I learned more about nautilus & gnome that I thought possible.

Thanks to all!

66 Comments:

  • Hey! Very Nice! Check out this website I found where you can make extra cash.
    It's not available everywhere, so go to the site and see if you can find something. I found something and make
    and extra $900 a month!

    http://www.degree-programs-online.info/extramoney.htm

    By Anonymous Anonymous, at 10:42 AM  

  • Thanks for all your work on this. Much appreciated.

    An Ubuntu user

    By Anonymous Anonymous, at 3:06 AM  

  • Greetings, I just wanted to say I appreciated your analysis of the nautilus startup and found it interesting to learn more about the entire thing.

    Hopefully they will fix some of this.

    Continue the excellent work.

    By Anonymous Anonymous, at 7:33 PM  

  • Thanks a lot for your work, i really enjoyed it, and hope it will help to make nautilus faster. thanks! :)

    By Anonymous a gnome user, at 1:03 PM  

  • Nice Work Phil...

    GIO

    By Anonymous Anonymous, at 1:46 PM  

  • Hello ezolt, This is a good article site.
    i found many information here.
    -----------------------------------------------
    http://global-in-arm.com
    Good luck, ezolt

    By Blogger Jonh Neo, at 9:13 PM  

  • Hey ezolt, Happy merry christmas.
    see you soon, ezolt

    By Blogger Jonh Neo, at 1:37 PM  

  • I congratulate all Soon Christmas
    Here some sites about the Christmases, a lot of interesting here
    [url="http://newyear.cjb.net"]new year celebration[/url]
    [url="http://here.is/christmasgift"]christmas gift[/url]
    [url="http://santaclaus2007.get2.us"]santa claus email[/url]
    [url="http://newyear.such.info"]new year[/url]
    [url="http://christmascard.ko188.com"]christmas card[/url]
    [url="http://christmasflower.ontheinter.net"]christmas flower[/url]
    [url="http://christmas.cx.la"]christmas[/url]
    [url="http://christmastree.dmdns.com"]christmas tree[/url]
    [url="http://www.fancyurl.com/christmasornament"]christmas ornament[/url]
    [url="http://christmassong.iceglow.com"]christmas song[/url]
    [url="http://www.irotator.com/rotator.php?u=happynewyear"]happy new year[/url]
    [url="http://chinesenewyear.isour.org"]chinese new year[/url]

    By Anonymous Anonymous, at 12:48 AM  

  • http://isater.com/news/categ-cancer_health_insurance-3708.html - cancer health insurance
    [url=http://isater.com/news/categ-cancer_health_insurance-3708.html] cancer health insurance [/url]

    By Anonymous Anonymous, at 10:06 AM  

  • By Anonymous Anonymous, at 2:02 AM  

  • Moreover, inquiry shows that on theprevious day a young girl had entered the household of the dreamer whohad pleased him, and who had given him the impression sex game freeware that she would notbe altogether opposed to an approach of this sort.The second half of the old school porno dream was discovered in the analysisto represent a flight from her husband, and the entering into intimaterelations with a third person, behind whom was plainly indicated Mr.To both of us it seemedpeculiar that the patient's mother thought nothing of the matter; ofcourse she herself alte frauen ficken must have been repeatedly in the situation describedby her child.I remember walking up Fifth Avenue with himone Sunday afternoon just after erotik ueber 40 he had shown me a letter from the manwho was then Comptroller of the Currency.

    By Blogger sink sink socks, at 11:07 PM  

  • "Lord, I have a problem!"
    "What's the problem, Eve?"
    "Lord, I know you've created me and have provided this beautiful garden and all of these wonderful animals and that hilarious comedy snake, but I'm just not happy."
    "Why is that, Eve?" came the reply from above.
    "Lord, I am lonely. And I'm sick to death of apples." "Well, Eve, in that case, I have a solution. I shall create a man for you."
    "What's a 'man,' Lord?"
    "This man will be a flawed creature, with aggressive tendencies, an enormous ego and an inability to empathize or listen to you properly, he'll basically give you a hard time. He'll be bigger, faster, and more muscular than you. He'll be really good at fighting and kicking a ball about and hunting fleet-footed ruminants, But, he'll be pretty good in the sack."
    "I can put up with that," says Eve, with an ironically raised eyebrow.
    "Yeah well, he's better than a poke in the eye with a burnt stick. But, there is one condition."
    "What's that, Lord?"
    "You'll have to let him believe that I made him first."
    :D :D :D

    _____________________________
    [url=http://shemalesex.iwannaforum.com]shemale bisexual[/url]

    By Anonymous Anonymous, at 6:08 PM  

  • Don't you tell anybody and I'll put you girl chatten on to exactly what grown-upgentlemen say and do when they go calling on the girls, he began.Some of them, like Professor Boris Sidis, reach at times conclusionswhich are sex freepics strangely similar to Freud's, but in their ignorance ofpsychoanalytic literature, they fail to credit Freud for observationsantedating theirs.His muse oftentook the direction of satire, but free teen sex video clip it was always good-natured even whenit hit the hardest.The negroes who had ridden with him were sprawled about him on theground; one of them lay sleeping, face hardcore porno zum kostenlosen download up, in the sunlight.

    By Blogger sink sink socks, at 11:16 PM  

  • By Anonymous Anonymous, at 8:34 PM  

  • By Anonymous Anonymous, at 5:44 PM  

  • And he had dropped into the habit of storing it about him,so that in any pocket dicke frauen fkk into which he plunged his hand he might find aroll of crisp evidence of reality.A general scrambletook place, and in another moment twenty grosse votzen ficken pairs of eyes, at least, werefeasting on the choice phrases lavished upon Mr.Afterwards the children kontakte anzeigen behindert are permitted to indulge themselvesin rational play, for I permit no amusement that is not alsoinstructive.I reckoned I'd bring Hiram round with me, said the young lady,lifting her searching eyes, after a pause, to the Colonel's, thoughhe was awful shy, and allowed that you didn't know him from Adam--oreven dicke frauen ficken free suspected his existence.

    By Blogger sink sink socks, at 11:14 PM  

  • WOW! Have you seen this site about cunt . It has great cunt pictures.

    By Anonymous Anonymous, at 1:06 PM  

  • By Anonymous Anonymous, at 4:38 PM  

  • He was going South, where his color wouldbe more considered than his reputation, and on the little local hechose there shemale movies free was a Jim Crow car--one, that is, specially set asidefor those of his race.Soon he would have to be making for what leathe western shore offered; but spermalive he was holding to the middle of theriver as long as he could, because with every mile the shores weregrowing more familiar, calling to him to make what speed he could.I told him a friend had commissionedme to muschi wixen make some inquiries about a cherished companion of his boyhoodnamed Leonidas W.It was not champagne or recreation gratis fotzenbilder that heneeded, not even a po'k-chop, although his desire for it had been asymptom, a groping for a too homeopathic remedy: he was homesick.

    By Blogger sink sink socks, at 11:08 PM  

  • [URL=http://equity-credit-lines.cresthomeloan.info/index.html]equity credit line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-information-line.html]credit equity home information line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-loan-low-mortgage-rate.html]credit equity home line loan low mortgage rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-home-line-loan-low-mortgage-rate.html]credit equity home home line loan low mortgage rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-does-equity-home-line-work.html]credit does equity home line work[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-illinois-line.html]credit equity home illinois line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-florida-home-line.html]credit equity florida home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/100-credit-equity-home-line.html]100 credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/bad-credit-equity-home-line-loan-low-mortgage-rate.html]bad credit equity home line loan low mortgage rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/bank-credit-equity-home-line-ohio-savings.html]bank credit equity home line ohio savings[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-michigan.html]credit equity home line michigan[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-tax.html]credit equity home line tax[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/125-credit-equity-home-line.html]125 credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-interest-line-rate.html]credit equity home interest line rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-refinance.html]credit equity home line refinance[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/citibank-credit-equity-home-line.html]citibank credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/bad-credit-equity-line.html]bad credit equity line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-question.html]credit equity home line question[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-line-loan.html]credit equity line loan[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-fargo-home-line-well.html]credit equity fargo home line well[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/best-credit-equity-home-line.html]best credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-texas.html]credit equity home line texas[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/california-credit-equity-home-line.html]california credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/bankruptcy-credit-equity-home-line.html]bankruptcy credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-virginia.html]credit equity home line virginia[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-equity-home-home-line-loan-vs.html]credit equity equity home home line loan vs[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-mutual-washington.html]credit equity home line mutual washington[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/bad-credit-credit-equity-home-line.html]bad credit credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/conventional-credit-equity-home-line-loan-other-vs.html]conventional credit equity home line loan other vs[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/calculator-credit-equity-line.html]calculator credit equity line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-fixed-home-line-rate.html]credit equity fixed home line rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-loan-versus.html]credit equity home line loan versus[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/best-credit-equity-home-line-rate.html]best credit equity home line rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-mobile.html]credit equity home line mobile[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-loan-vs.html]credit equity home line loan vs[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/calculator-credit-equity-home-line.html]calculator credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-line-rate.html]credit equity line rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-loan.html]credit equity home line loan[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line-rate.html]credit equity home line rate[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-line.html]credit equity line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/credit-equity-home-line.html]credit equity home line[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info]equity credit lines[/URL]
    [URL=http://equity-credit-lines.cresthomeloan.info/sitemap.html]Equity Line of Credit[/URL]

    By Anonymous Anonymous, at 1:08 AM  

  • MESSAGE

    By Anonymous Anonymous, at 9:05 PM  

  • By Anonymous Anonymous, at 9:05 PM  

  • lolita chin
    lolitas 13
    lolitas kid
    free anal sex
    lolita picture
    lolita suck
    lolita cp terra loltobbs ftp lol
    lolita vip lolitas virgins bbs
    lolita lolly fuck movie children
    [url=http://shelterland.com/index.shtml]lolita chin[/url]
    [url=http://shelterland.com/index1.shtml]lolitas 13[/url]
    [url=http://shelterland.com/index2.shtml]lolitas kid[/url]
    [url=http://shelterland.com/index3.shtml]free anal sex[/url]
    [url=http://shelterland.com/index4.shtml]lolita picture[/url]
    [url=http://shelterland.com/index5.shtml]lolita suck[/url]
    [url=http://shelterland.com/index6.shtml]lolita cp terra loltobbs ftp lol[/url]
    [url=http://shelterland.com/index7.shtml]lolita vip lolitas virgins bbs[/url]
    [url=http://shelterland.com/index8.shtml]lolita lolly fuck movie children[/url]

    By Anonymous Anonymous, at 10:11 AM  

  • MESSAGE

    By Anonymous Anonymous, at 12:51 PM  

  • By Anonymous Anonymous, at 12:51 PM  

  • By Anonymous Anonymous, at 9:26 PM  

  • Watiti.com
    Join me and my circle of friends at http://www.watiti.com, an online social networking community that connects people from all over the world.

    Meet new people, share photos, create or attend events, post free classifieds, send free e-cards, listen music, read blogs, upload videos, be part of a club, chat rooms, forum and much more!

    See you around! Bring all your friends too!

    Watiti.com

    By Anonymous Anonymous, at 7:56 AM  

  • Miss Jane is poundingaway at one of her long music pieces, sex trailer movie and it won't be over before youhave time to get to Rutgers and back again.Among the symbols which are usedin this manner free blonde anal sex in germany there are of course many which regularly, or almostregularly, mean the same thing.Such avariety of scenery, such an opportunity for mom walks and rambles! But,William, if you could only make up your mind not to----Oh, that is all right! exclaimed Podington.My present aachen porno suggestion is simply the great extension ofthe system, so that all public machine-work may be done by it.

    By Blogger sink sink socks, at 11:44 PM  

  • By Anonymous Anonymous, at 12:18 AM  

  • [url=http://ambien-free.blogspot.com/]ambien[/url] http://ambien-free.blogspot.com/
    Good Luck!

    By Anonymous Anonymous, at 6:35 AM  

  • Watiti.com
    Join me and my circle of friends at http://www.watiti.com,
    an online social networking community that connects
    people from all over the world.

    Meet new people, share photos, create or attend
    events, post free classifieds, send free e-cards,
    listen music, read blogs, upload videos, be part of a
    club, chat rooms, forum and much more!

    See you around! Bring all your friends too!

    Watiti.com

    By Anonymous Anonymous, at 10:48 AM  

  • By Anonymous Anonymous, at 3:01 PM  

  • preteen underage video could feel him rising, and could feel his hands on her preteen 8 15 pic as he guided her to lay on her stomach.
    Shifting her position, Jack pulled on her hips, raising her ass up into the air,
    his hands carressing the cheeks of her buttocks and then grasping her hips firmly, his hardness pressing once more into her warm folds.
    She moaned as he slammed suddenly into her, the motion starting a chain reaction that continued as he violently pounded into her willing cunt.
    His fingers gripped into her hips harshly, pulled her toward him, and as he thrust into her she felt herself propelled in the other direction,
    only to be pulled quickly back. Her breasts swinging freely, preteen angels felt a tingling pain in her nipples as her breasts bounced in time to his thrusts.
    She moaned as she felt him tense, arching her back and trying to further expose her sex, trying to take as much of him into her as possible.
    To the side, in the mirror, she could see his figure above her, his head tilted back as he groaned his pleasure.
    Emily.... Oh, God... Emily!
    "Come, Jack... Please, Come..." preteen hidden pic pleaded, and she found that the words were more for her than for encouragement.
    She wanted to feel the hot rush of his seed, wanted to drink it inside herself and hear his moans of pleasure.
    Emily... Emily...
    He slammed almost painfully into her, and preteen sex pic galleries found herself nearly screaming outloud at the wonderful abuse.
    He filled her so completely that she felt nearly impaled; stretched her so intensely that she feared to be torn.
    His fingers dug into her flesh, and she could hear him slapping against her as his moans began to rise.

    By Anonymous Anonymous, at 2:35 PM  

  • Here all free
    [url=http://free-editor.pozitor1.info] free editor [/url]
    [url=http://free-one.pozitor1.info] free one [/url]
    [url=http://free-sample.pozitor1.info] free sample [/url]
    [url=http://free-personals.pozitor1.info] free personals [/url]
    [url=http://free-bingo.pozitor1.info] free bingo [/url]
    [url=http://free-wallpaper.pozitor1.info] free wallpaper [/url]
    [url=http://free-sudoku.pozitor1.info] free sudoku [/url]
    [url=http://free-ware.pozitor1.info] free ware [/url]

    By Anonymous Anonymous, at 10:56 AM  

  • By Anonymous Anonymous, at 12:04 PM  

  • Watiti.com
    Join me and my circle of friends at http://www.watiti.com,
    an online social networking community that connects
    people from all over the world.

    Meet new people, share photos, create or attend
    events, post free classifieds, send free e-cards,
    listen music, read blogs, upload videos, be part of a
    club, chat rooms, forum and much more!

    See you around! Bring all your friends too!

    Watiti.com

    By Anonymous Anonymous, at 7:35 PM  

  • Watiti.com
    Join me and my circle of friends at http://www.watiti.com,
    an online social networking community that connects
    people from all over the world.

    Meet new people, share photos, create or attend
    events, post free classifieds, send free e-cards,
    listen music, read blogs, upload videos, be part of a
    club, chat rooms, forum and much more!

    See you around! Bring all your friends too!

    Watiti.com

    By Anonymous Anonymous, at 9:55 PM  

  • Hi all!
    It is very interesting site. I like it
    Look it:
    [url=http://12351.freegoodies.nl/forum_an.php?forum=12351]free+hardcore+sex+video[/url]
    free+hardcore+sex+video
    http://12351.freegoodies.nl/forum_an.php?forum=12351
    [url=http://pinkworld1.21.forumer.com]pinkworld[/url]
    pinkworld
    http://pinkworld1.21.forumer.com
    [url=http://coupleerotic.jconserv.net]couple+erotic+fetish+guide+sex[/url]
    couple+erotic+fetish+guide+sex
    http://coupleerotic.jconserv.net
    [url=http://12350.freegoodies.nl/forum_an.php?forum=12350]free+sex+trailer[/url]
    free+sex+trailer
    http://12350.freegoodies.nl/forum_an.php?forum=12350
    [url=http://gaymovie.21.forumer.com]gay+porn+movie[/url]
    gay+porn+movie
    http://gaymovie.21.forumer.com
    [url=http://sexchat.jconserv.net]free+sex+chat+rooms[/url]
    free+sex+chat+rooms
    http://sexchat.jconserv.net

    By Anonymous Anonymous, at 10:11 AM  

  • She loved her brother so much. He was older than her, 23, and she was just 19. they got along great and did most everything together. free bbs lolita preteens took a year off of school and was going to go away to the collage a few hundred miles away in a big city. Their town was small and Kitty had never lived in a big city. She was scared to move there all by herself. She especially was afraid to lose her brother.
    "You aren't losing me, Kitty, I will be right here, right here waiting for you to return. I will call you and write to you if you like..."
    She interrupted Jarod,
    "Of course I want to keep in touch! I wish dad didn't need your help on the ranch, I wish you could come with me."
    free bdsm personals pouted and lowered her head. Her brother, Jarod, lifted it with one finger and smiled,
    "If you keep that lip out there a bird might land on it and shit in your mouth..."
    They both laughed. She always laughed so much around him. Kitty hugged him tight and he held her close, they both wanted to stay in the moment forever. However, dad called down the stairs and they finished their morning rituals before they went out to take care of the animals.
    She was already dressed and was in the bathroom brushing her teeth as her brother got dressed in his room. He took his boxers off while the door was cracked slightly open and Kitty peeked at him between the door to his bedroom and the frame. All she could see as a well toned, tight as hell ass. She had seen him naked when they where younger and experimenting with the world, but he used to be chubby. Now he was sculpted, as was she, from working hard on the ranch.
    She ran the toothbrush on her teeth slower as she stared and soon was just standing there sucking on the end of it. He pulled his jeans up and threw on a t-shirt and she backed away into the bathroom to finish. As she was washing her face he came up behind her, pressing his body against her back in order to reach in front of her to get his own toothbrush. His crotch pressed firmly on her ass and she felt a little tingle go through her body. She never thought of it as sexual but it made her a little damp. This was her brother, somebody whom she loved, not a dirty man off the streets bent on getting in her pants. The thought of a twisted stranger trying to fuck her made her smile and her nipples got hard. She was normally a very dirty girl in bed. (and very rarely did she use a bed at all!)
    free big tit videos smiled down at his sister while she dried her face and he put toothpaste on his toothbrush. He could smell her free hardcore teen sex, it smelled like strawberries. He loved that smell. Sometimes he would use her shampoo so he could smell strawberries all night. She did the same thing. Kitty would sometimes put a dab of his cologne on her neck so she could smell his scent all night. They never told each other this, or anybody else for that matter. People just wouldn't understand.
    They went out early in the morning before breakfast and started chucking hay from a truck into the horse pasture. It was misty out and a loud thunder grew in the hills beyond. It was the beating of many hoofs. A few whinnies and snorts where heard as the horses drew closer. Soon their black outlines where seen galloping and throwing their heads up and down and side to side as they came. Kitty drove the truck off a few yards away from the sinuous mass of pure strength and wild hearts as they closed in on the food. The stallion trotted around the food, watching his mares and foals carefully, then he pushed into the crowd to eat with them. The matriarch of the group was pushing her way in next to the stallion and they nuzzled each other gently then they continued eating.
    Kitty and Jarod smiled as they drove off to the barn to clean up the loose hay off the ground that they spilled from the bails. As they worked, Jarod told his sister dirty jokes he read online,
    "A guy comes home from work feeling bad about what he had done there. He lays and ponders his actions for a while. A voice in his head tells him, 'it is okay, a lot of doctors have sex with their patience.' He starts to feel a little better then another voice in his head said, 'yes, but you are a veterinarian!'"
    They both laughed, though he tried to keep a straight face to add to the humor. Kitty grabbed a handful of loose hay and threw it at him. He ducked and grabbed some of his own, tossing it at her. They laughed and threw hay, getting covered in it and soon she jumped on his back as he went to get more. He spun them both around and they fell to the ground in a pile of the yellow hay laughing and wrestling.
    free black sex chat pinned Kitty down under him and they stopped laughing. They where breathing hard and their hearts where pounding. They looked into each other's green eyes and smiled shyly. They had just transformed into virgin lovers, touching the opposite sex for the first time. All the thoughts of being apart drifted away and they where in the moment. Nothing else existed but them, they had each other.
    He bent his head down and kissed her softly on the lips. Jarod pulled his head back to make sure she wasn't going to protest and she lifted her free lolitas teenie galleries head up and kissed him back. He slid his hands under her and held her tight as they started kissing heavier, passionately and fervently. Their tongues explored, each in the other's mouth. It was unlike any kiss either of them had ever had. This was pure passion, love and trust.
    Jarod moved his hands down and pulled her shirt up, moving his fingers to the front to touch her firm, round, perky tits. Her nipples where rock hard as he touched them. Her breasts where so soft and smooth. Kitty arched her back slightly and slid her own hands up his shirt, feeling his bare chest. He was hot and she could feel his heart pounding, hers was, too. Their hearts thumped away in their chests so hard – like the beating of the wild horse's hooves on the morning damp ground.
    They managed to pull each others' shirts off between heavy kisses and he deftly undid her bra from the front clasp. Her tits fell out and her hard nipples stared up at him. He lay his body on top of hers slightly so she could just feel his skin on her nipples. It drove free interracial sex wild. Jarod pressed his hips against hers and pushed his hardening cock into her wet pussy through their pants.
    He started to unbutton her jeans as he kissed a path from her lips to her jaw, across her collarbone and between her tits. He licked and sucked each pink nipple, nibbling at times. Then he made his way down further to her tummy and hips. When he got down there he had her pants unbuttoned and pulled halfway down her upper thigh. He kissed and bit her hip bones and looked at her light brown fur patch that lay peeking over the underwear and jeans that he was sliding off.

    By Anonymous Anonymous, at 7:21 AM  

  • Then, impressed with a vibrator silicon horror of the glaring apparition, he gatheredhimself up, and with a bound dashed along the tow-path.As defendant's counsel are furnished with copiesalso, they cannot plead 'surprise,' as in the introduction of newmatter, and as plaintiff's counsel relies evidently upon the jury'sattention to his opening, he would klingeltoene zum down not be the first person to distractit.He thought at first that this picture of his home was gay piss scat some new anddelicate device put forth by his press-agent.Miss Van Kamp would not for the world have it understoodthat this unconventional meeting, made allowable by her wrenchedankle, could possibly fulfill the functions of a formal fickt euch alle introduction.

    By Blogger sink sink socks, at 2:36 AM  

  • MESSAGE

    By Anonymous Anonymous, at 5:50 AM  

  • http://invest-options-online.usabestlink.info - invest options online

    http://option-investing.usabestlink.info - option investing

    [URL=http://invest-options-online.usabestlink.info/index.html]invest options online[/URL]
    [URL=http://option-investing.usabestlink.info/index.html]option investing[/URL]
    [URL=http://option-investing.usabestlink.info/investing-in-future-and-option-market.html]investing in future and option market[/URL]
    [URL=http://option-investing.usabestlink.info/lease-option-investing.html]lease option investing[/URL]
    [URL=http://option-investing.usabestlink.info/fear-investing-option-without.html]fear investing option without[/URL]
    [URL=http://option-investing.usabestlink.info/investing-lease-option-turnkey.html]investing lease option turnkey[/URL]
    [URL=http://option-investing.usabestlink.info/lease-option-real-estate-investing.html]lease option real estate investing[/URL]
    [URL=http://option-investing.usabestlink.info/investing-in-option.html]investing in option[/URL]
    [URL=http://option-investing.usabestlink.info/investing-in-stock-option.html]investing in stock option[/URL]
    [URL=http://option-investing.usabestlink.info/stock-option-investing.html]stock option investing[/URL]
    [URL=http://option-investing.usabestlink.info/future-in-investing-market-option.html]future in investing market option[/URL]
    [URL=http://option-investing.usabestlink.info/investing-lease-option.html]investing lease option[/URL]
    [URL=http://option-investing.usabestlink.info/fear-investing-option-without.html]fear investing option without[/URL]
    [URL=http://option-investing.usabestlink.info/investing-lease-option-turnkey.html]investing lease option turnkey[/URL]
    [URL=http://option-investing.usabestlink.info/estate-investing-lease-option-real.html]estate investing lease option real[/URL]
    [URL=http://option-investing.usabestlink.info/in-investing-option.html]in investing option[/URL]
    [URL=http://option-investing.usabestlink.info/investing-option.html]investing option[/URL]
    [URL=http://option-investing.usabestlink.info/in-investing-option-stock.html]in investing option stock[/URL]
    [URL=http://option-investing.usabestlink.info/investing-option-stock.html]investing option stock[/URL]
    [URL=http://option-investing.usabestlink.info/investing-in-future-and-option-market.html]investing in future and option market[/URL]
    [URL=http://option-investing.usabestlink.info/lease-option-investing.html]lease option investing[/URL]
    [URL=http://option-investing.usabestlink.info/fear-investing-option-without.html]fear investing option without[/URL]
    [URL=http://option-investing.usabestlink.info/investing-lease-option-turnkey.html]investing lease option turnkey[/URL]
    [URL=http://option-investing.usabestlink.info/lease-option-real-estate-investing.html]lease option real estate investing[/URL]
    [URL=http://option-investing.usabestlink.info/investing-in-option.html]investing in option[/URL]
    [URL=http://option-investing.usabestlink.info/investing-in-stock-option.html]investing in stock option[/URL]
    [URL=http://option-investing.usabestlink.info/stock-option-investing.html]stock option investing[/URL]
    [URL=http://option-investing.usabestlink.info/invest-options-online.html]invest options online[/URL]

    By Anonymous Anonymous, at 5:50 AM  

  • [url=http://fradalitassociates.co.uk/logs/airline-ticket.html]airline ticket[/url]

    By Anonymous Anonymous, at 9:58 AM  

  • Hello, old Poopoo, you grow personal; and if you can't keep a civiltongue in your head, www you must go out of my counting-room.Then asickle was lying on the board; next a scythe was added; and, finally, www hebeheld the likeness of an old peasant mowing the grass in front of theboy's distant parental home.Itwas this that caused him to graze an oath with less www margin that he hadallowed himself in twenty years.But look here, William, exclaimed Buller, www it will get there just aswe do; no horse could stand a roaring up in the air like that!Podington laughed.

    By Blogger sink sink socks, at 12:13 AM  

  • He leaned back comfortably in his chair, and f_n half closing his eyes as in dreamy reminiscence, said, slowly: Your reply, Mr...

    By Blogger sink sink socks, at 7:17 PM  

  • The decomposition ofthese mixed images by analysis is www often the quickest way to aninterpretation of the dream.For though he was good-natured, he was very shiftless, and it was, asour national proverb says, like www pulling teeth to teach him.And then you will be able to repeat your remarks www to me in thestreet , continued the Colonel, bowing, as he persisted in followinghis visitor to the door.These reduced me at once to submission, and I am www almostashamed to confess that, either through pain or vexation, there came afew tears into my eyes.

    By Blogger sink sink socks, at 12:14 AM  

  • By Anonymous Anonymous, at 8:43 AM  

  • By Anonymous Anonymous, at 8:00 AM  

  • Hey, while searching for widgets for my blog, I stumbled upon www.widgetmate.com and wow! I found what I wanted. A cool news widget. My blog is now showing latest news with title, description and images. Took just few minutes to add. Awesome!

    By Anonymous Anonymous, at 12:30 AM  

  • Cool Sites!
    http://buy-valium.awardspace.com
    http://adult.wtcsites.com
    http://rain.prohosting.com/tireshop
    http://christmas-nightmare.freewebspace.com
    http://cheaplaptop.traficexperts.com
    http://toothwhitening.searchtrafficwizard.com
    http://puntacanavacation.searchtrafficwizard.com
    http://hotelvenice.searchtrafficwizard.com
    http://dragonballz1.freewebspace.com
    http://cingular.741.com
    http://southbeachdiet.741.com
    http://airfare1.white.prohosting.com
    http://myspacegenerator.250free.com
    http://cosmetology.bookst.info
    http://bluetoothheadset.0catch.com
    http://freewebs.com/allergytreatment
    http://cosmetology-school.blogggest.com
    http://freewebs.com/evanrachelwood
    http://hertzcarrental.250free.com
    http://buy-tramadol-1.100webspace.net
    http://cheaplaptop.freewebspace.com
    http://toothwhitening.freewebspace.com
    http://puntacanavacation.freewebspace.com
    http://hotelinvenice.freewebspace.com
    http://cheapfuton.freewebspace.com
    http://ionamin.freewebspace.com
    http://ccart.freewebspace.com
    http://travelcard.1sweethost.com
    http://rhodiola.angelcities.com
    http://poker1.funpic.de
    http://m.domaindlx.com/gamb
    http://transcred.freehostia.com
    http://www.dubhot.port5.com
    http://kitchenmixed.tripod.com
    http://alarmsys.proboards102.com
    http://construct1.funpic.de
    http://homeins.freehostia.com
    http://floorhardwood.bravehost.com
    http://balancecred.bravehost.com
    http://dragonballz1.741.com
    http://vendingcandy.tripod.com/
    http://pentcredit.angelcities.com
    http://lumberwood.tripod.com/
    http://dragonballz.usaroyalcasino.com
    http://megasexshop.bka.ru
    http://dragonballz2.funpic.de
    http://suzukigrandvitara.tripod.com
    http://cardonation.tripod.com
    http://www.freewebs.com/enterprisecar
    http://budgetcar.741.com

    By Anonymous Anonymous, at 7:25 AM  

  • c8X8UD Your blog is great. Articles is interesting!

    By Anonymous OnlinePharmacy, at 12:08 AM  

  • muVKTk Please write anything else!

    By Anonymous buy phentermine cheap pharmacy buy phentermine c, at 10:58 AM  

  • T2LYNd Please write anything else!

    By Anonymous name, at 12:00 PM  

  • ZnxG4q Magnific!

    By Anonymous purchase meridia online, at 12:11 PM  

  • actually, that's brilliant. Thank you. I'm going to pass that on to a couple of people.

    By Anonymous castroville texas motels, at 12:28 PM  

  • Wonderful blog.

    By Anonymous name, at 12:08 PM  

  • Thanks to author.

    By Anonymous charlie daniels band tour schedule history, at 12:57 PM  

  • Magnific!

    By Anonymous atomic tour race, at 7:31 AM  

  • Nice Article.

    By Anonymous graduate student loans, at 11:05 PM  

  • Thanks to author.

    By Anonymous ringtones, at 2:20 AM  

  • Good job!

    By Anonymous , at 6:14 AM  

  • tKR4P2 Thanks to author.

    By Anonymous inlevitra, at 11:21 AM  

  • Please write anything else!

    By Anonymous free ringtones for, at 11:50 AM  

  • To be a upright charitable being is to procure a amiable of openness to the in the seventh heaven, an cleverness to trust aleatory things beyond your own pilot, that can lead you to be shattered in hugely outermost circumstances for which you were not to blame. That says something uncommonly outstanding with the prerequisite of the honest compulsion: that it is based on a corporation in the up in the air and on a willingness to be exposed; it's based on being more like a shop than like a treasure, something fairly fragile, but whose extremely precise attraction is inseparable from that fragility.

    By Anonymous Anonymous, at 11:12 AM  

  • meet my new site -

    [url=http://trailfire.com/codeine] street names for codeine [/url]

    http://trailfire.com/codeine
    [url=http://trailfire.com/codeine] promethazine with codeine [/url]

    By Anonymous Anonymous, at 7:33 AM  

  • A likeable noachian majority is the prize of a well-spent youth. As a substitute for of its bringing wretched and low prospects of decay, it would hand out us hopes of eternal lad in a recovered world.

    By Anonymous Anonymous, at 5:25 AM  

Post a Comment

<< Home