-1

Preciso inserir um dado com mais de 50 milhões de linhas que está no s3 numa tabela do hbase. Estou utilizando o EMR da AWS para usar o cluster com os serviços do hadoop como o hbase. Já consegui colocar os dados do s3 no hdfs e preciso passar esses dados para a tabela do hbase. O mapreduce ImportTsv não era eficiente para transportar uma grande carga de dados, então descobri que tenho que usar o bulk load. Estou utilizando este tutorial para isso: https://www.nodalpoint.com/bulk-load-data-to-hbase-in-oracle-big-data-appliance/

O job do 'org.apache.hadoop.hbase.mapreduce.ImportTsv' que é usado para converter os arquivos HDFS em um storefile é finalizado com sucesso, por mais que eu obtenha estes erros no ambiente de execução:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/hbase/hbase.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollin                                                                                        gFileAppender.java:223)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:3                                                                                        07)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.j                                                                                        ava:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.j                                                                                        ava:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigura                                                                                        tor.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigura                                                                                        tor.java:768)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyC                                                                                        onfigurator.java:648)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurato                                                                                        r.java:514)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurato                                                                                        r.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionCon                                                                                        verter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.apache.log4j.Logger.getLogger(Logger.java:104)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.jav                                                                                        a:262)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:1                                                                                        08)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct                                                                                        orAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC                                                                                        onstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(Log                                                                                        FactoryImpl.java:1025)
        at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementat                                                                                        ion(LogFactoryImpl.java:844)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactory                                                                                        Impl.java:541)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactory                                                                                        Impl.java:292)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactory                                                                                        Impl.java:269)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.apache.hadoop.hbase.mapreduce.ImportTsv.<clinit>(ImportTsv.java:8                                                                                        1)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/hbase/SecurityAuth.audit (Permission den                                                                                        ied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollin                                                                                        gFileAppender.java:223)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:3                                                                                        07)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.j                                                                                        ava:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.j                                                                                        ava:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigura                                                                                        tor.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigura                                                                                        tor.java:768)
        at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyC                                                                                        onfigurator.java:672)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurato                                                                                        r.java:516)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurato                                                                                        r.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionCon                                                                                        verter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.apache.log4j.Logger.getLogger(Logger.java:104)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.jav                                                                                        a:262)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:1                                                                                        08)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct                                                                                        orAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC                                                                                        onstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(Log                                                                                        FactoryImpl.java:1025)
        at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementat                                                                                        ion(LogFactoryImpl.java:844)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactory                                                                                        Impl.java:541)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactory                                                                                        Impl.java:292)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactory                                                                                        Impl.java:269)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.apache.hadoop.hbase.mapreduce.ImportTsv.<clinit>(ImportTsv.java:8                                                                                        1)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFAS].

Então, seguindo com o tutorial, para importar os dados de fato para o hbase eu preciso rodar esse comando 'hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /storefile flat_table_sf10', mas sempre obtenho o erro:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/hbase/hbase.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.apache.log4j.Logger.getLogger(Logger.java:104)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:108)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
        at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.<clinit>(LoadIncrementalHFiles.java:118)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFA].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/hbase/SecurityAuth.audit (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
        at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.apache.log4j.Logger.getLogger(Logger.java:104)
        at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:108)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
        at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.<clinit>(LoadIncrementalHFiles.java:118)
log4j:ERROR Either File or DatePattern options are not set for appender [DRFAS].

Gostaria de entender como resolver esses erros ou se tem algum método mais eficiente de executar essa tarefa.

1 Resposta 1

0

sera que ajuda?

Primeiro, você precisa organizar esses dados em um arquivo HFile. O arquivo HFile tem um formato específico que contém os dados a serem carregados. O exemplo a seguir mostra o formato do arquivo HFile para esses dados:

John Doe,32,123 Main Street

Em seguida, você pode executar o comando Bulk Load como mostrado abaixo para carregar os dados do arquivo HFile na tabela 'users':

hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles '/caminho/para/arquivo/HFile' 'users'

1
  • Mas pra criar um HFile não é com esse job como no tutorial? hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns="HBASE_ROW_KEY,Closing:Price,Current:Price" -Dimporttsv.bulk.output="/user/oracle/data/stocks/storefile" Stocks /user/oracle/data/stocks/stocks.txt Commented 24/02/2023 às 1:06

Você deve fazer log-in para responder a esta pergunta.

Esta não é a resposta que você está procurando? Pesquise outras perguntas com a tag .