Monday, 3 August 2020

SQOOP Export Example : HDFS to MySQL using SQOOP Export


Target table must exists in the Database. So we are creating the empty table

mysql> create table retail_db.customers_exported as select * from customers where 1 = 0;

// Now the record count is zero for customers_exported table of retail_db;

hdfs dfs  -ls /user/cloudera/customers
Found 9 items
-rw-r--r--   1 cloudera cloudera          0 2020-08-02 21:58 /user/cloudera/customers/_SUCCESS
-rw-r--r--   1 cloudera cloudera     118147 2020-08-02 21:57 /user/cloudera/customers/part-m-00000
-rw-r--r--   1 cloudera cloudera     119075 2020-08-02 21:57 /user/cloudera/customers/part-m-00001
-rw-r--r--   1 cloudera cloudera     119057 2020-08-02 21:57 /user/cloudera/customers/part-m-00002
-rw-r--r--   1 cloudera cloudera     118831 2020-08-02 21:57 /user/cloudera/customers/part-m-00003
-rw-r--r--   1 cloudera cloudera     118798 2020-08-02 21:57 /user/cloudera/customers/part-m-00004
-rw-r--r--   1 cloudera cloudera     119294 2020-08-02 21:57 /user/cloudera/customers/part-m-00005
-rw-r--r--   1 cloudera cloudera     119892 2020-08-02 21:58 /user/cloudera/customers/part-m-00006
-rw-r--r--   1 cloudera cloudera     120431 2020-08-02 21:58 /user/cloudera/customers/part-m-00007


sqoop export \
-connect "jdbc:mysql://localhost/retail_db" \
-username root \
-password cloudera \
-table customers_exported \
-export-dir /user/cloudera/customers \
-input-fields-terminated-by ',' \


mysql> select * from retail_db.customers_exported limit 10;
+-------------+----------------+----------------+----------------+-------------------+--------------------------+---------------+----------------+------------------+
| customer_id | customer_fname | customer_lname | customer_email | customer_password | customer_street          | customer_city | customer_state | customer_zipcode |
+-------------+----------------+----------------+----------------+-------------------+--------------------------+---------------+----------------+------------------+
|        4665 | Jessica        | York           | XXXXXXXXX      | XXXXXXXXX         | 3982 Middle Horse Line   | Escondido     | CA             | 92025            |
|        4666 | Mary           | Smith          | XXXXXXXXX      | XXXXXXXXX         | 340 Velvet Barn Gate     | Manati        | PR             | 00674            |
|        4667 | Cheryl         | Gonzalez       | XXXXXXXXX      | XXXXXXXXX         | 5375 Iron Cloud Mews     | Caguas        | PR             | 00725            |
|        4668 | Mary           | Schwartz       | XXXXXXXXX      | XXXXXXXXX         | 4263 Old Robin Mount     | Mechanicsburg | PA             | 17055            |
|        4669 | Robert         | Smith          | XXXXXXXXX      | XXXXXXXXX         | 4905 Lazy Stead          | Humacao       | PR             | 00791            |
|        4670 | Sharon         | Bailey         | XXXXXXXXX      | XXXXXXXXX         | 3890 Jagged Apple Dell   | Caguas        | PR             | 00725            |
|        4671 | Eric           | Diaz           | XXXXXXXXX      | XXXXXXXXX         | 281 Middle Blossom Ledge | Chandler      | AZ             | 85225            |
|        4672 | Mary           | Smith          | XXXXXXXXX      | XXXXXXXXX         | 7800 Tawny Creek Trace   | Detroit       | MI             | 48235            |
|        4673 | Wayne          | Hodges         | XXXXXXXXX      | XXXXXXXXX         | 7509 Iron Concession     | Caguas        | PR             | 00725            |
|        4674 | Mary           | Lynn           | XXXXXXXXX      | XXXXXXXXX         | 7694 Velvet Turnabout    | Jamaica       | NY             | 11434            |
+-------------+----------------+----------------+----------------+-------------------+--------------------------+---------------+----------------+------------------+

No comments:

Post a Comment

Flume - Simple Demo

// create a folder in hdfs : $ hdfs dfs -mkdir /user/flumeExa // Create a shell script which generates : Hadoop in real world <n>...