StartNew, ContinueWith will default to TaskScheduler.Current, Current will return the Default scheduler, When not called from within a task(MSDN).

To avoid the default scheduler issue, you should always pass an explicit TaskScheduler to Task.ContinueWith and Task.Factory.StartNew.

ContinueWith is Dangerous

Is the ContinueWith() Task guaranteed to run on my scheduler? If not, which scheduler will it use?

No, it will use the scheduler passed to the original Task. ContinueWith will default to use TaskScheduler.Current, in which case is the default thread-pool task scheduler. There is no propagation between your provided context to task.Start and the one used inside the continuation

From the source:

public Task ContinueWith(Action<Task> continuationAction)
{
    StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller;
    return ContinueWith(continuationAction, 
                        TaskScheduler.Current, 
                        default(CancellationToken),
                        TaskContinuationOptions.None, ref stackMark);
}

@Noseratio - read it, but still skeptic about the validity of this behavior - I ran the first task on a non-default scheduler for a reason. Why did TPL decide the continuation, which is always sequential to my task, should run on another?

I agree - this is not the best design - but I imaging that defaulting to TaskScheduler.Current is there for ContinueWith to be consistent with Task.Factory.StartNew, which defaults to TaskScheduler.Current too, in the first place. Stephen Toub does explain the latter design decision:

In many situations, that’s the right behavior. For example, let’s say you’re implementing a recursive divide-and-conquer problem, where you have a task that’s supposed to process some chunk of work, and it in turn subdivides its work and schedules tasks to process those chunks. If that task was running on a scheduler representing a particular pool of threads, or if it was running on a scheduler that had a concurrency limit, and so on, you’d typically want those tasks it then created to also run on the same scheduler.

Thus, ContinueWith uses the current (ambient) TaskScheduler.Current of whatever task is currently executing at the moment you call ContinueWith, rather than the one of the antecedent task. In case this is a problem for you and you cannot explicitly specify the task scheduler, there is a workaround. You can make your custom task scheduler to be the ambient one for a particular scope, like this:

using (var scheduler = new MyKickAssTaskScheduler())
{
    Task<Task> outer = new Task(() => 
    {
       Task foo = new Task(() => DoSomething());
       foo.ContinueWith((_) => DoSomethingElse());
       foo.Start(); // don't have to specify scheduler here
       return foo;
    }

    outer.RunSynchronously(scheduler);
    outer.Unwrap().Wait();
}

Note that outer is Task<Task>, hence there's outer.Unwrap(). You could also do outer.Result.Wait(), but there's some semantic difference, especially if you used outer.Start(scheduler) instead of outer.RunSynchronously(scheduler).