Dear Microsoft...

[daddykatt.livejournal.com]

... I understand that Excel 03 can only process ~250k lines. This is why we have 2007.

Your claim that 2007 can process an unlimited number of lines was why we updated.

So, when I call you and ask you why, when I import plain-ol tab-delimited text, that Excel STOPS after a million lines, your answer of "Because that's all it can process" does not bloody well compute.

so, can it process a million lines or can it process an unlimited number of lines?

Riddle me that.

And, yes, I actually need to, routinely, process well over a million lines of data. And no, I can not split the data.

Comments

[ljquin.livejournal.com]

For: "2007 can process an unlimited number of " read, "we never actually tested and hoped no-one would ever try importing large data sets".

[tragnon.livejournal.com]

Access or SQL server...

[vortex.livejournal.com]

Ditto!...

[melstav]

Or *ANY* RDBMS. Hell, even MySQL would be a better choice than to try using excel as a database.

[xforge.livejournal.com]

Totally this. In fact, I'll take ANY excuse - ANY - to not have to use Excel for ANY reason EVER. My God what a nightmare that thing can be.

[daddykatt.livejournal.com]

we are not using excel as a DB - thank gods - but in order to modify mass amounts of data from our Oracle DB, we export it to a tab-delimited file and then import into excel... because we need to modify hundreds of thousands of lines with the same data - And the company will not allow/support/install/whatever any software other than MS office and the DB program that we use.

[taiki.livejournal.com]

Here's a tip.

PHP, python and anything else that can talk to your oracle DB would probably not have this limit. :) Thoughta bout any other alternatives?

[daddykatt.livejournal.com]

Oh yeah, we have THOUGHT about it... but not having any control over ANYTHING kinda limits ones options

[taiki.livejournal.com]

You have complete control over it, it's something you would code. ;)

[firon.livejournal.com]

how about using Access to handle it? Either by getting Access to talk directly to your database, or by importing the CSV data to it? Certainly not ideal, but might work better than excel.

[trixtah.livejournal.com]

Actually, Logparser (http://www.codinghorror.com/blog/archives/000369.html) is a fantastic tool for parsing large delimited text files. Run SQL statements against big fat files! I think it's the best thing that MS have ever written. I can run through several 10s of gigs worth of mail logs counting up how many messages were delivered to a specified recipient in about 3 minutes.

But not being able to use a DB at all for processing that kind of stuff is screwed. I'd also agree that sneaking a P* scripting language onto a box somewhere for pre-processing would make sense. But maybe they'd be happier with Logparser since it is an MS product.

[ogphil.livejournal.com]

i must say, i do not know the feeling, but until the very end of this i thought you were just pushing their envelope. they should be doing that, anyway.
i hope they get the message.

[cheezemeister-x.livejournal.com]

There is no computer on the planet that can process an unlimited number of lines, in any piece of software. If you're in IT, you should know that. :)

[mouser.livejournal.com]

Bah - FIFO queue it. As long as the computer runs (and it can keep up) you can do SOME processing...

...not spreadsheets, though. Need a more defined definition of "process" for that.

[anivair.livejournal.com]

Of course there isn't, but there is no sense telling users that it's unlimited when you mean anything other than "limited by ram and hard drive space". Plenty of programs can process that much data, and if Excel can't process whatever you've got, they have no business saying they can.

[preserver3.livejournal.com]

I'd argue that there are computers, given infinite maintenance, that can handle unlimited lines... ;^)

[wxgeek.livejournal.com]

Jesus friggin' Christ, why in god's name are you using Excel to process a million lines of data?

[cjkline83.livejournal.com]

I agree with this comment.

[preserver3.livejournal.com]

I love spreadsheets and like Excel, but I also like AWK, SED and Perl. Over 150K lines or so... um, it's time to move to AWK.

In other words, I second this comment that second the first comment.

[wxgeek.livejournal.com]

I agree with this comment.

[daddykatt.livejournal.com]

We have an oracle database, but the frontend program that we need to work it will not allow us to make mass changes (IE adding the dame string of data to the same field for 1.67 million entries in the DB)

The only way that the company will allow to do this is to use excel because they will not install/support/allow any new software

normally excel works for this because most of the datasets are a few thousand lines... no more than 100k

this one just happened to be friggin huge

[wxgeek.livejournal.com]

I hate your life for you.

[jokergirl.livejournal.com]

...whoever thought this was a good idea and why didn't anyone stop him?