loading huge sql files into godaddy

Posted by sergio_101 on Feb 1st, 2008
2008
Feb 1

the problem

for the past year or so, we have been suggesting that all of our clients use godaddy for web hosting. for as long as we have been suggesting them, we have not had any problems with them.

the one beef i have always had with them is that it is a huge pain to load up a mysql database from a sql file. for pretty much all of our sites, we use typo3 as our content managment system. this system produces a fairly large sql file. one of the tables in that file (a caching table) is way over 2mb. this means that we can’t split it up to upload it to phpmyadmin. the other problem is that while the control panel says that it supports zip, i can’t, for the life of me, get it to work.

the almost solution

in the past, what i have done is upload the sql file to a remote directory, then get on the phone with godaddy tech support and BEG them to load up the file. this is a really big pain for everyone involved, as there is alot of naming schemes and questions that everyone has to agree on before anything can happen. they are extremely cautious when doing this (as well they should be), so this can take an incredible amount of time to plow through.

after about a day or so, i finally end up with something workable, and we can move on.

the final solution

after going through this many times, i remembered a command in php: “system();“.

this command allows you to feed a string into the command line, and have it evaluated. you are then returned whatever the response to the command is..

so, all you have to do is make a php file with something like this in it:

[sourcecode language=’php’]

'; //in this line, the DBNAME is the name of the database you are populating //the databse.sql is the file you uploaded .. change those names to something //that makes sense $command = "mysql DBNAME < datbase.sql" $last_line = system($command, $retval); // Printing additional info echo '
Last line of the output: ‘ . $last_line . ‘
Return value: ‘ . $retval; ?>

[/sourcecode]

so, here’s how you do it:

  • upload your sql file to a unique directory. make it a temp directory, so you can just delete the whole thing when you’re done.
  • upload the file with the php code above to the same directory.
  • using your web navigator, go to the php page you created. this will automatically run your code.
  • take note of anything weird that shows up in the output, but chances are, everything will go just fine.
  • delete the entire directory, to make sure no one accidentally overwrites anything.

some notes

in most cases, when running mysql from the command line, you have to add “-u username -p password” to authenticate you as a user. with godaddy’s hosting, simply running “mysql” logs you in with the correct credentials. if you try to specify credentials manually, it will flop.

hope that was helpful to someone out there!

PHP and the PayPal API - from 10,000 feet..

Posted by on Jan 27th, 2006
2006
Jan 27

i spent a good deal of time plodding through the documentation to get this thing set up, only to find that the documentation is spread all over the site, and you really have to pull pieces from everywhere.. and alot of them are not 100% correct.. and alot of them don’t fit together..

the biggest problem that the doco has is that there is no view from 10,000 feet. the views are all from the ground.. here are some hints to save you some aggravation..

this is not a comprehensive “how-to” doc.. but rather a gotcha doc.

  • the first thing you do is create accounts in what is called a “sandbox.” you need a developer’s account to do this, but in essence, you are just creating a buyer and a seller account. that’s it. there is no juju here.
  • while you are doing this, you also need to generate fake certs.. write down the passphrase you use. you will follow the codebases and doco to hell and back. by the time you have assembled a workable codebase, you will have forgotten the passphrase you use. i did. on a good note, it’s easy to generate a new one. when you generate a new one, the old one is IMMEDIATELY taken out of service. if you are working on a live cert, there is a risk of disaster on a high volume site..
  • next, you need to install the pear files. for those of you who are not familiar with pear, it is alot like CPAN in the perl world. my guess is that if you are struggling with this, you are probably using this on a shared system. there are special rules for setting this up on a shared system. mostly, this involves setting up paths to the pear libraries. make SURE you install the libraries above your web root so that no one can abuse them..
  • now that you have all that setup, there is a web console that you can use to test your api calls. although this looks interesting and fun, i spent a good amount of hours getting all the path dependencies worked out. after a good afternoon chasing these down, i nixed the whole idea. it would be easier to code bareback here.
  • you will most luck at this point if you just TOTALLY dump the whole set of examples that paypal gives you and go here. there is a good codebase there, BUT keep in mind, there are some problems with it.. when you start running it, and it goes haywire, look for an xpm tag. remove it AND its mate. also, there is a whacked out form tag in there somewhere. move form and its twin OUTSIDE the whole form (they are embedded kinda weird. dreamweaver will bitch like mad about this). fix that first before you go insane.
  • ditch ALL of the POST variable initializations. trust me on this one. there are a TON that don’t really do anything but make you go back and forth about a million times trying to figure out what gets used and what doesn’t. this is all stuff that should be done in javascript OR left completely out of a tutorial on the API.
  • create session variables for your info. the data is passed twice. this is a typo minefield. i found a neat way to do this (after my eyes glazed over for the millionth time tracking down POST variables.) you can see how i did it (and will do it from now on) . let me know if anyone has any comments on this. on my way out he door today, i started wondering if mabye i had a switch on in PHP that allowed you to just call $_SESSION and $_POST variables at will. i remember hearing that you could do this under some configurations. this seems almost as dangerous as putting your weiner in a meat grinder.
  • finally, run some fake transactions. go back to that link on pear. there is a bogus credit card number on there that you will have erased when you set your forms up. you need that because you will have forgotten the fake credit card number of your sandbox account. i spent about ten minutes trying to figure out how to get that back, but i gave up. that other credit card will do. by now, you will have made a bogus express transfer anyway, so you know your sandbox account works.

well, that’s about it. in retrospect, it wasn’t THAT bad.. but it was sort of like those little chinese finger cuffs. once you figure out how to get them off, it all makes sense.. but when you are jammed all up in there, it’s maddening.. let me know if this helps..

2006
Jan 26

here’s a problem i consistently run into..

i am implementing a paypal service that takes the form of three files..

a file to gather the payment data…

a file to process the data, and make sure it is correct..

a file to provide api calls to paypal to process the payment..

the api samples do this the way most every other php script does..

via POST, they send the data to the next form..

in the receiving form, there are some agonizingly LONG strings of the following:

$firstName=$_POST['firstName']; $lastName=$_POST['lastName']; …and so on for ALL variables…

keep in mind that if you proceed this way, ANY time you update your inintial form, you will have to update ANY references to the $_POST call. this sucks..

i dug around for a bit.. and found this..

the first thing i did was take my $POST variables, and map them to $_SESSION variables. i thought for SURE this was alot tougher than i thought. it’s way easier.

session_start(); $_SESSION=$_POST;

will copy the array, and initialize the session variables..

since i used javascript to gussy the data up, there was really nothing to do on this end but to display the data and let the user go back and fix it if necessary..

here’s where it gets cool…

once we get to the api calls, we would, in all other cases, be required to initialize our variables again.. one by one.. based on the $_POST variable.. this SUCKS..

here’s a neat way to skate by this …

there is a command called session_encode() that will serialze your session variables..

once you serialze them, you can just unserialze them using unserialize() and you will in effect initialize all your $_SESSION variables as normal variables…

here’s how i did it:

$vars=session_encode(); print $vars; // do this once, just to see it work.. unserialize($vars);

at this point, you will see that you now have all your variables initialized..

keep in mind that if you change them, it has nothing to do with $_SESSION.. but in my case.. that wasn’t the gist of the thing.. all i needed was to make API calls with them without having to reference them via $_SESSSION..

in other words..

i can use:

$IPAddress

rather than

$_SESSION['IPAddress']

i thought that was pretty neat…

getting MacLeo to run on OSX

Posted by on Jan 10th, 2006
2006
Jan 10

note.. i could not add the link to the title here, it looks like this site is banned by b2evolution.. just google for “install macleo”..

it took m a bit to figure it out, but DUH..

i finally got leo (a literate programming tool) to run on OSX…

keep in mind that the install info asks you to install the file and a script as such:

!/bin/sh

$2 only works if .leo is one of the extensions.

/usr/bin/pythonw ~/leo/src/leo.py $2

but make sure you chmod+x the file that you write, otherwise platypus will let you make the file, but it will not run MacLeo, nor will it tell you why it just craps out. the deal is, it has no +x privs.

chmod+x the file, and you are good to go.

one thing that i thought was interesting about the whole thing is that the mac version of leo was pretty much that hardest one to install..

i hope it’s worth it…

Literate Programming

Posted by on Jan 9th, 2006
2006
Jan 9

i had no idea about this concept until i met with scott yesterday. i am going to start looking into this alot more seriously, i think, since it seems to be a place where all i need to spend more work.. in devloping my models and the documentation for them. my biggest weakness is that i like to get in and start writing, withouth thinking much about the consequences of the future. in smaller projects, this is not that big of a deal, but i am finding that as projects get larger and larger, or more obtuse, i am having more trouble remembering where i was in the mix. so, for 2006, i am going to start working on this..

literate programming.. taste great.. less filling.. well.. more filling.. but a good kinda filling..

Scott Collins’ Journal

Posted by on Jan 8th, 2006
2006
Jan 8

scott collins - blog imagei spent 12 hours programming with scott yesterday. although i always thought i was a pretty good and pretty thoughtful programmer, i have never met anyone who was as a good a programmer as scott. without getting too wrapped up in the details, i think what i gleaned most from working with scott was that i have been trying to throw alot of heavyweight at alot of lightweight problems. i have spent way too much time trying to figure out how to make the tools i have fit the problem at hand. working with scott, he had the opposite approach. fit the problem at hand to the tools. i seem to do this alot, since i can remember. i spend too much time trying to come up with a sophisticated or elegant solution to a problem. in the meantime, the problem gets lost in the mud of the solution. i thought it was cool that while scott acknowledges that he is one of the heaviest heavyweight programmers, this is due to the fact that this is ALL he’s done his whole life. i found him a good teacher.. and a good listener.. and a good dude..